Domopalooza 2024: Magic ETL Beta Enhancements

For years, Magic ETL has been Domo’s secret weapon to building data pipelines with ease. With a simple drag-and-drop interface, Magic ETL makes it easy to clean and combine data without knowing SQL. Magic is great for shaping and building pipelines once your data is in Domo, but we also know that the challenge facing many organizations today is making an assortment of data tools work in harmony to create a rich data foundation.

Today, we’re excited to announce a new set of Beta features to Domo’s Magic ETL product that will enhance the experience for all users.

First, the “Join Data” tile in Magic ETL has been updated to include a freeform expression box that can be used to define the join relationship.  This is an optional experience and can be used instead of the current experience of defining the relationship using the drop-down menus. This opens the ability for users to write their own freeform “ON” clause for their joins and gives flexibility to write more complex join expressions, including non-equijoins and ranged joins. With this new update, friction is removed when joining data sets in Magic ETL. Access this feature in the “Join Data” tile in Magic ETL.

Next, “Upsert” is now an option for a data update type in the Output Tile configuration in Magic ETL. This allows users to select an upsert key on their output tile, which will result in output rows with that same key being updated in the existing output data set. Output rows with a key that doesn’t already exist in the output data set will be appended. Upsert in Magic ETL allows the customer to update rows in their output data set without reprocessing the full data sets, based on a unique, row-level key.  This replaces any of the remaining needs for recursive dataflows. This pipeline pattern is more efficient because Domo is not reprocessing data or performing complex comparison logic to update an existing data set. Access this feature on the “Output Dataset” tile in the “Update Method” drop-down menu.

Finally, we are excited to announce the combined power of Magic ETL + Cloud Amplifier with Snowflake. With data held in so many clouds, accessing and integrating is becoming more difficult. Cloud Amplifier is a no-compromises experience, delivering the full power of Domo on top of any data structure.  Domo Cloud Amplifier is the tissue that connects these disparate systems, allowing data architects to get the most out of each data source. We know the effort needed to create a data foundation in your warehouse, and we want to honor that investment. Now, beginning with Snowflake, and to be followed by other cloud warehouses, we are bringing Magic ETL directly to your data source for intelligent, governable, and faster transformations right where your data lives. Experience the Magic ETL advantage without moving your data.

Contact your CSM to get started.