Flat Package Editor Dmg

By: Ron Kirchgessner | Updated: 2014-08-29 | Comments (8) | Related: More >Integration Services Data Flow Transformations

UUByte DMG Editor is a handy tool for making bootable Mac USB. More importantly, it supports Windows OS and macOS at the same time. Wait for 10-15 minutes, a macOS installer USB is ready for repairing your Mac and leaving your personal data on Mac untouched. Open macOS DMG files on Windows. Extract any file from a DMG archive in just a few clicks. 30 day money back guarantee Expert support for 1 year. Previously, if you wanted to distribute one of these packages, you’d have to put the package inside a DMG or tar it up. Another interesting thing happened during the evolution of OSX packages. PackageMaker, a wart in the eyes of many developers and source control systems, was officially deprecated and dropped in Xcode 4.6 (and there was much.

Problem

SQL Server Integration Services (SSIS) can import data from a fixed widthflat text file (i.e. no field delimiters), using the Flat File Source component.When importing this type of file, you must set the field names, field widths,and starting positions ahead of time. Typing in this information when there areonly a handful of fields is not too burdensome, but when there are hundreds offields, manually configuring and validating each field can take a long time.Here is a workaround using an existing import table with table columns alreadydefined. Thank Tim Katheeder for coming up with this workaround.

Let’s move to the Packages tabs. The package’s Settings tab. Remember when we started our new installer project, I pointed out that Packages is the title for the left sidebar of the app? Since a project can be configured to build multiple distributables, each one has a yellow/brown icon next to its name. Open Composer and authenticate locally. Select the package source you want to build as a DMG from the Sources list in the sidebar. In the toolbar, click Build as DMG. Select a location to save the package and click Save.

NOTE: I use the term 'field' when referring to positional data in the flattext file (e.g. field Name consist of character positions 1 to 10). The term 'column' is used when referencing columnar data in a SQLServer table (e.g. column Nameis varchar (10)).

Solution

Background

For one project I worked on, the data came from an old COBOL system. Each rowof data had 3,500 characters organized into 530 fields. The text file was 100MB. So yes, we are talking about a substantial amount of typing to specify thefields to import and defining the table columns where the data will reside.

My preparation consisted of two activities:

Flat Package Editor Dmg Tutorial

a) copying the first few rows ofthe 100 MB data file into a test file for import development, and

b) using Excelto convert the flat file design specifications into a SQL statement creating animport destination table.

In the example presented here, I created a test filehaving 3 rows of data with 35 characters representing 5 fields, the designspecifications for the test file, and a create table statement based on the testfile design specifications.

Example: Test text file

Example: Test file design specifications

Example: Import destination table structure

About the Import Destination Table

This tip does not cover how to translate the fixed width flat text filedesign specifications into a SQL create table query as there are a variety ofapproaches depending on your particular situation. However, there is one detailthat merits special attention. For this fixed width flatfile import process to work, the import destination table columns areall defined as text (e.g. VARCHAR). Keeping in mind the source system,COBOL may export a numeric field defined as 's9(6)v99' that is represented by an 8character field. The 'v' in the format is referred to as an implied decimal.This means a flat file field value of '12345678' (8 characters) translates to atable column value of '123456.78' (9 characters). When importing a fixed widthflat text file, the field widths must be exact or subsequent fields may receivean incorrect value. By loading all fields as text the individualpeculiarities can be addressed on a case-by-case basis. Transforming data fromthe Import destination table structure to the final table structure is astandard SSIS data transformation.

Example: Import destination table with raw data (ImportText)

Example: Final table structure

Example: Final table with formatted data (ImportFinal)

We've described the data we are starting with and how we want the datarepresented when finished. Here are the steps to make it happen.

Step 1: Create a Data Flow Task

Create a Data Flow task in SSIS to house the workaround.

Step 2: Create and configure an OLE DB Source

a) Double click the Data Flow Task to build the Data Flow.

b) Create an OLEDB Source component.

c) Double click the component to open the OLE DB SourceEditor and configure the Connection Manager to point to the import destinationtable (ImportText).

When an OLE DB Source component is configured, SSIS stores the table'sinformation in the component for reference. Now the import destination tablecolumns (ImportText) are available for reference.

Step 3: Create a Flat File Destination

a) Create a Flat File Destination component.

b) Draw a Data Flow Path betweenthe OLE DB Source component and the Flat File Destination component. Do not openthe Flat File Destination component until after the Data FlowPath is established.

After the Flat File Destination component is connected to the OLE DB Source,SSIS automatically copies the column information from the OLE DB Sourcecomponent (ImportText) to the Flat File Destination component. In other words,the Flat File Destination component now has ImportText table specifications.

Note: The red circle X in the Flat File Destination component indicates thecomponent configuration is incomplete and will be addressed in the next step.

Step 4: Configure the Flat File Destination

a) Double click the Flat File Destination component to open the Flat FileDestination Editor.

b) Click on the New… button and the Flat File Format dialogwill open.

c) Select the Ragged Right option and click OK.

d) Back in the FlatFile Connection Manager Editor, type in 'f' as File name, and then close theeditor.

e) In the Flat File Destination Editor click on Mappings and you willsee the Flat File Destination component configured with the import tablespecifications. Click OK.

The Ragged Right option is used in step 4c because it is not unusual for datafiles with large record structures to have varying record lengths. Step 4D is alittle misleading as we needed to complete the Flat File Destination componentconfiguration. Once the configuration is done, the OLE DB Source importdestination table columns (ImportText) are loaded into the Flat File Destinationcomponent. The 'f' is not an actual file and was used for configuration purposesonly.

Step 5: Create the Flat File Source

a) Create a Flat File Source component.

b) Delete the Data Flow Path betweenthe OLE DB Source component and the Flat File Destination component.

c) Draw aData Flow Path between the Flat File Source component and the Flat FileDestination component. Do not open the Flat File Source component until afterthe Data Flow Path is established.

d) Open the Flat File Source Editor componentto confirm the columns are populated.

Completing step 5d means the Flat File Source component now has the fielddefinitions from the Flat File Destination component courtesy of SSIS automaticimport. In other words, the ImportText table column definitions are now the Flat File Source component field definitions.

Note: The red circle X in the Flat File Destination component indicates thecomponent configuration is incomplete due to changing the data source from theOLE DB Source to Flat File Source component.

Step 6: Create and configure the OLE DB Destination

a) Create an OLE DB Destination component.

b) Delete the Data Flow Pathbetween the Flat File Source component and the Flat File Destination component.

c) Draw a Data Flow Path between the Flat File Source component and the OLE DBDestination component.

d) Double click the component to open the OLE DBDestination Editor.

e) Configure the OLE DB Destination component ConnectionManager to point to the import destination table (ImportText). Select Mappingsto confirm the input and destination columns are mapped correctly.

f) Click OKto save the configuration changes.

With the completion of step 6f, the Flat File Source and OLE DB Destinationcomponents are compatible and ready for use. Both components are using the testfile design specifications.

NOTE: If you get this warning, you will need to start over as these stepsneed to be completed in one session.

Step 7: Cleanup

The OLE DB Source and Flat File Destination components are no longer neededand can be deleted. The Data Flow Task is ready to import data from a flat fileto the defined table.

The imported data is in a text format in the import destination table(ImportText). Transferring the data to the final table structure (ImportFinal)is a typical SSIS transformation activity and not covered here.

Next Steps

Now that you have easy access to the data, the next step is cleaning andtransforming the raw text data. Here are some helpful tips on making thathappen.

About the author

Around the time of the release of OS X Mountain Lion, Xcode moved to a single drag ‘n drop .dmg model to simplify the user installation experience, and Apple was nice enough to make the Command Line tools a separate download. Unfortunately, this hasn’t simplified the process of mass deployment, and we now have more moving pieces to keep track of than ever before. Anyone who’s deployed Xcode recently may be familiar with its laundry list of post-installation tasks.

Some downloads, like the Command-Line Tools and earlier iOS Simulator/SDK versions, show up in a new “Components” download area located in Xcode’s Preferences. We’ll look at where this index comes from, how we can inspect it to get the iOS simulator .dmg download URLs, and one method of modifying the simulator installer packages so that they install to the correct location via any standard package distribution method like Munki or Casper. If you manage installing the Command-Line Tools as well, you’ll find metadata here that will help with tracking version installs (anyone who’s tried to manage deploying/updating them knows they don’t use Apple package versioning).

Xcode Preferences: Downloads

We’ll also quickly review the other steps typically required to “finalize” the Xcode installation for most deployment scenarios. Big thanks to Nate Walck for testing what I’d originally documented for the iOS Simulator installation and determining that I’d skipped an important step!

Deploying Xcode via a software management mechanism such as Munki, Casper or ARD usually requires a few steps for a fully-functional install. These usually include:

• copy the Xcode.app bundle from the .dmg to /Applications
• copy out the MobileDevice.pkg (and MobileDeviceDevelopment.pkg as of Xcode 4.5) from Xcode.app/Contents/Resources/Packages and install them separately
• assuming your users are not admins, adding an appropriate user group to the _developer group, and running /usr/sbin/DevToolsSecurity -enable, which handles modifying the security policies in /etc/authorization
• optionally install the Command Line Tools for the appropriate OS version
• optionally accept EULAs and configure the options for downloading extra components and documentation, via the defaults domain at com.apple.dt.Xcode

Installing these additional components has been already documentedinseveralplaces, but I hadn’t yet seen anyone describe how they deployed versions of the iOS Simulator/SDK, which normally would be an optional component download.

Xcode seems to automatically include the most current version of the SDK in its .app bundle, so this is primarily useful if you want to be able to deploy older versions as well. But if you manage Xcode it’s a good idea to become familiar with this index file anyway.

I recently discovered via the Charles web proxy tool what Xcode actually downloads from Apple to populate its list of additional components for download, which is also used to determine how they’re installed. It turns out these indexes are also cached locally on the client, so no web traffic sniffing is even needed.

Once you’ve launched Xcode at least once as a user, head over to that user’s Xcode cached downloads folder at ~/Library/Caches/com.apple.dt.Xcode/Downloads.

In mine, I have some cached downloads (which Xcode helpfully renamed to saner filenames than it stores on its downloads site), and a few different index files. Each one is from a different version of Xcode. On this machine I’ve gone from version 4.3 to 4.4. to 4.5, and I believe these index filenames remain the same for patch versions. The only one getting updated now is the last one dated Nov 13, f9556a…dvtdownloadableindex.

These cached versions are binary plists, so either first convert them to the XML plist format or open them an editor like TextWrangler or TextMate, which support transparent conversion of binary plists. If we open this up, we can see it serves a purpose similar to Apple’s Software Update .sucatalog files, except all its logic and metadata is inline. One item to take note of is the source key at the bottom:

Flat Package Editor Dmg Example

The GUID in this URL seems to be unique to the Xcode major version, as is the SHA-1 that makes up the filename stored in the local cache. A couple useful header values from the web server for this URL:

Mac Flat Package Editor

The ETag in this case is an md5 of the file and the time it was modified in the Unix time format. Given this, you could monitor changes to either of these headers to know when there may be new changes in the index. (Of course, this index URL would likely change when there’s a new major release of Xcode. At the time of writing the most recent version is 4.5.2, but inside the index there is at least one update restricted to only the preview release of Xcode 4.6).

The meat is in the downloadables array, so let’s look at one dict entry from it:

Lots of useful, readable information here. We see the ‘displayed name’ Name key, the identifier and version, the .dmg download at source, and some logic for how Xcode determines it’s installed via InstalledIfAllPathsArePresent.

Notice the RequiresADCAuthentication key, which is fairly new. Previously these legacy Simulator downloads were hosted at http://adcdownload.apple.com, and would prompt you for an Apple ADC login. Currently there aren’t any downloads that require an ADC login, so Apple’s likely just reserving this behaviour for future use. It means it’s easier for us to actually grab the .dmg without needing an auth cookie set by our browser, which used to require a manual download of Xcode in order to have it set.

The other very important piece here is the InstallPrefix key, set to $(DEVELOPER). We’re going to need this to actually install the .dmg correctly.

Once we’ve downloaded and mounted the installer .dmg, we can see it’s just a .pkg installer. The installer is a flat package, and we can use the pkgutil to expand it to a working directory on our desktop and take a look:

We’ve got the most basic flat package structure possible: a Bom, PackageInfo and Payload file. We can use the lsbom command to get a list of all the payload items. If we take a look at the first few items, we see the top few items in a folder hierarchy:

Flat Package Editor Dmg Software

Those ‘./’s at the beginning look like relative paths, don’t they? They are. If we’d install this package now using the installer command and specify a root OS volume like /, we’d wind up with this Platforms folder at the root of our drive, which is not what we want. If we dig around inside the Xcode app bundle, we can find where the more current iOS Simulator is stored, at:

Flat Package Editor Dmg Online

This correlates with the InstallPrefix key we saw earlier, telling us that we should effectively prefix this package’s target location with folder with /Applications/Xcode.app/Contents/Developer/. To do this, we’ll edit the PackageInfo file that contains metadata about the package. Specifically, we want to set the install-location attribute in the pkg-info element. See Stéphane Sudre’s excellent Flat Package Format page for more info on flat packages. With our small modification, the PackageInfo should now look something like this:

Note the less-than-useful version number in the pkg itself. Now we can compile this back to a new flat package using the --flatten option for pkgutil:

The resulting file, iPhoneSimulatorSDK-5.0.0.1.pkg, should now be installable as an addition to your Xcode installation.

Unfortunately, we’re still not done. If we skim through the index for this SDK’s identifier, we come across another entry, that contains a PatchFor key:

Again, the logic is clear. It’s a patch for version 5.0.0.1 of Xcode.SDK.iPhoneSimulator.5.0, and Xcode will know it’s installed according to the contents of the InstalledIfAllSHA1SumsMatch key. In this case, it’s the SHA-1 hashes of a bunch of library files. If you inspect the BOM of this patch installer, you’ll see it’s only installing the same patched libraries given in this InstalledIfAllSHA1SumsMatch dict. So, to install this you’ll need to perform the same modification to the PackageInfo file to set the install-location attribute.

It’s important to apply this patch, because if it is not applied, Xcode may detect that an installed component is not fully up to date, and prompt a dialog at launch, similar to when the MobileDevice support package is not installed or the correct version:

A user can skip through this by failing the authorization prompt, but there’s otherwise no option to simply bypass the update.

With that, we should have everything we need to deploy a specific legacy iOS Simulator/SDK version. We know where to get the installers, what version they are, how Xcode knows whether they’re installed, and information for dependencies and patches that may see new uses in future versions. Your software distribution mechanism should be able to use all of this metadata to manage installations and updates for Xcode. If it doesn’t, you may need to supplement it with some of your own checking mechanisms.

This post’s specific use case of iOS Simulators may seem esoteric, but it’s a good example of the index metadata Xcode uses for its supplemental downloads, and a practical use of the pkgutil command-line tool. With more and more packages being distributed in the flat package format, being familiar with using it to audit and modify packages is essential for administering OS X.

• Developer Binaries on OS X, xcode-select and xcrun –
• Deploying Xcode - The Trick With Accepting License Agreements –
• A Tour of Charles, Your HTTP(S) Swiss Army Knife –
• Building native extensions since LLVM 5.1 –