Batch Processing with pro Fit

Table of contents

• Introduction
• Methods for batch processing
• Method 1: Batch processing from a program
• Method 2: Batch processing from an Apple Script
• Notes
• Further reading

Introduction

The techniques described here can be used for various applications where a large number of data files must be processed in some way. You may want to combine the data of all files into a single file. Or you may want to open each file, calculate the average of each column, and create a new data file that contains the average values of all data files.

In this TechNote we discuss the following example:

You have a large number of files, each file containing the data of an exponential decay. You want to fit the data from each file to pro Fit's built-in exponential function. The results of all these fits are to be collected in a new data file.

The techniques we present for solving this problem can easily be applied to any other type of batch processing.

This TechNote and all example files can also be downloaded as a binhexed file (16 kB).

Methods for batch processing

1. You open all the files by hand (select them all in the finder and double-click). Then run a pro Fit program that goes through all open data windows and runs a fit for each of them.
2. You write an Apple Script that runs through all the files. For each file, the script tells pro Fit to open it, then calls a pro Fit program for fitting and collecting the fit results, and then closes the file again.
3. You write an external module that uses the Macintosh File Manager for cycling through all the files. The module opens each file, runs a fit, stores the fit results, and closes the file again.

In all the examples presented here, we will fit the data in each file to pro Fit's built-in function "Exp" by varying the parameters "A", "t0" and "const". The parameter "x0" is to be set to 0 and held constant (because "A" and "x0" are mathematically redundant and they cannot be fitted simultaneously).

Method 1: Batch processing from a program

The short program that follows does all the work for us.

program MultipleFit;

var wind, myDataWindow;
    windowCount;

begin
 NewWindow(dataType);     {create a new window for storing results}
 myDataWindow := FrontWindow;  {save a reference to it}

 windowCount := 0;
 wind := NextWindow(myDataWindow);      {get next window behind it}
 while (wind <> 0) do                   {cycle through all windows}
 begin
   if GetWindowType(wind) = dataType then   {if the window is data}
   begin
     windowCount := windowCount+1;
     Writeln('Fitting window ', windowCount);
     SetCurrentWindow(wind);          {use the data of this window}
     SetFunctionParam('Exp', 1, 1);       {set starting parameters}
     SetFunctionParam('Exp', 2, 0);        {these values depend on}
     SetFunctionParam('Exp', 3, 1);          {your model and data}
     SetFunctionParam('Exp', 4, 0);
     Fit('Exp',1,2,0,0, false);                       {run the fit}
     SetCurrentWindow(myDataWindow);   {window for storing results}
     if NumFitParams <> 0 then          {if the fit was successful}
     begin
       data[windowCount,1] := FittedParams(1);        {get results}
       data[windowCount,2] := FittedParams(3);  {we did not fit x0}
       data[windowCount,3] := FittedParams(4);
     end;
   end;{if}
   wind := NextWindow(wind);                    {go to next window}
 end;
end;

To cycle through the data windows, the program uses the commands NextWindow and GetWindowType. NextWindow returns a reference ID for a window that lies behind another window. GetWindowType returns the window type, i.e. tells if a window is a data window, drawing window or text window. When the program finds a data window, it makes it the "current data window" (by calling SetCurrentWindow), then sets some reasonable starting values for the function Exp (by calling SetFunctionParam), runs the fit (by calling Fit), switches back to the window with the fitted data (by calling SetCurrentWindow(myDataWindow) and stores the results there.

Method 2: Batch processing from an Apple Script

It might be more convenient to have a method that fits all files in a given folder. There is no way to do this from a program within pro Fit, but there is a way to do this from an Apple Script.

To use Apple Scripts, you need System 7.5 or later (or System 7.1 with the Apple Script extension installed). To define an Apple Script, you need a script editor, such as Apple's Script Editor. This editor comes with System 7.5 and should be located in the folder "Apple Extras" on your system hard disk.

To define the Apple Script, run your script editor, open a new window and enter the following:

-- bring up a dialog for selecting the folder
set myFolder to choose folder with prompt "Choose a folder with data files:"

-- create a list with all files in the folder
set myFiles to list folder myFolder -- a list of files in myFolder
set myFileCount to count myFiles -- the number of files in myFolder

-- now start fitting with pro Fit
tell application "pro Fit"
  activate -- bring pro Fit to front
  set error alerts to false -- disable error reports within pro Fit
  make new table -- open a data window for storing our results
  set name of front window to "Result data" -- and set its name
  set globalData 0 to 0 -- this is our window counter
  repeat with i from 1 to myFileCount
    set theFile to item i of myFiles -- get the i-th file
    try
      open file ((myFolder as string) & theFile) as table -- open file
      write line "processing: " & theFile
      run program "SingleFit" -- run the program in pro Fit
      close window theFile saving no -- close without saving
    on error errText
      write line "cannot process: " & theFile & " (" & errText & ")"
    end try
  end repeat
  set error alerts to true -- enable error reports within pro Fit
end tell

Note that you can save your script as a "compiled script" from your script editor. The script can then be opened as any other pro Fit module using the "Load Module" command in the Customize menu. Or you can put it in the "pro Fit modules" folder. In this way the script is always accessible inside pro Fit and you don't need anything else to do your batch processing.

Before running the script, you must define the program "SingleFit". To do this, switch to pro Fit and define the following program:

program SingleFit;
begin
  globalData[0] := globalData[0]+1;           {increase window counter}
  SetCurrentWindow(FrontWindow);     {use data of front window for fit}
  SetFunctionParam('Exp', 1, 1);              {set starting parameters}
  SetFunctionParam('Exp', 2, 0);               {these values depend on}
  SetFunctionParam('Exp', 3, 1);                  {your model and data}
  SetFunctionParam('Exp', 4, 0);
  Fit('Exp',1,2,0,0, false);                              {run the fit}
  SetCurrentWindow(GetWindowID('Result data'));    {window for results}
  if NumFitParams <> 0 then                 {if the fit was successful}
  begin
     data[globalData[0],1] := FittedParams(1);            {get results}
     data[globalData[0],2] := FittedParams(3);      {we did not fit x0}
     data[globalData[0],3] := FittedParams(4);
  end;
end;

It uses globalData[0] as a counter for the fitted windows.

Notes

1. Both above examples set the starting parameters of the function Exp before fitting. The values used here may have to be varied depending on the data sets to be fitted.
2. The pro Fit package comes with an Apple Script called "batch processing". This script can be found in the folder "Examples:AppleScript" - it shows some other tricks that can be done with Apple Scripts. The use of an Apple Script might appear a bit clumsy at first sight because it requires to write the script in a separate application. But Apple Scripts are a very powerful tool, and once you get used to their weird syntax, there's nearly nothing you cannot do with them.
3. If you use an Apple Script often, you may want to add it to pro Fit's Prog menu. To do this, save the Apple Script as a "compiled script" from your script editor, then switch to pro Fit and choose "Load Module" from the Customize menu to load the script into pro Fit. To add the script permanently to pro Fit, save it as a "compiled script" and place it into the folder "pro Fit Modules" in the pro Fit folder.

Further reading