Creating Synthetic Data Sets

<< Statistics`LinearRegression`
<< Graphics`Graphics`

Many times in modeling, we need to simulate data sets. This can happen for a variety of reasons. We may have a proprietory data set, and we need to create a data set with the same properties of this data set that can be realeased to the public. You can see this if you read the information for authors for The Machine Learning Journal. Proprietory data sets could contain information about a product that is not public, military information, or personal information. Even if your data set is not proprietory, you may want to create a group of data sets which have similar properties for use in the classroom.

Also, using simulated data sets gives you some method of estimating the errors in the parameters in your model directly. 

Besides which, it is fun.

The True Reality Underlying Function: Mother Nature's True Function

Underlying the model, there is Mother Nature's True Function, the function that is guiding the creation of the data set you measure. Of course, random errors, measuring inaccuracies, and (hopefully not large) systematic errors make it impossible for you to realize Mother Nature's True Function exactly.

For example, say some physical process results in a linear relation between t and f. The
parameters in the relation, a and b, are Mother Nature's True Parameters. For our discussion, we will say that the following function is Mother Nature's True Function, and Mother Nature's True Parameters are precisely a = 3.45 and b = 0.067. There is absolutely no error in Mother Nature's True Parameters.

a  =  3.45;
b = 0.067;
ftrue[t_]  =  a  +  b t;

Without the error that intrudes on any experiment, if we measured the system at t = 8, we would expect our measurement to be exactly 3.986.

ftrue[8]

Experimental Measurement of the System

You will forgive me if I don't conduct an experiment to get the data; instead, I will pretend that this first initial run is me doing some experiement or collecting some data from somewhere which forms my data set. The data0 represents, for us, the actual data set you have. The Random call represents the experimental error that is introduced when the experiment is performed. The SeedRandom makes sure that I get the same random number sequence each time I evaluate the cell.

Below, I create the experimental realization of Mother Nature's True Function, by collecting data and fitting (using Mathematica's Fit, now that we know what it means) a linear equation to the data.

SeedRandom[4];
Clear[data0]
data0  =  
        Table[{t, ftrue[t] + Random[Real, {-2, 2}]}, {t, 1, 
            200}];

f[t_]  = Fit[data0, {1, t}, t]
Regress[data0, {1, t}, t, 
      RegressionReport -> {RSquared, 
          EstimatedVariance}]

dataplot  =  
        ListPlot[data0, 
          DisplayFunction -> Identity];
modelplot  =  
        Plot[f[t], {t, 0, 200}, 
          DisplayFunction -> Identity];
Show[dataplot, modelplot, 
      DisplayFunction -> $DisplayFunction]

We see that the experminetal realization of Mother Nature's True Function is

f (t) =  3.58798  +  0.0662225 t

Creating Hypothetical Data Sets When the Distribution Is Known

Since we know the underlying distribution of the data points (the Random call I added), we can create a number of new data sets, and obtain estimates of the parameters a and b from them. Then we can get estimated errors in these parameters. 

We are not getting new experimental values--we are simulating the experiement based on the fact that we know that underlying process that produced the data.

Here is one simulated data set. The different SeedRandom means that I get a different sequence of random numbers than when I "did the experiment" above.

SeedRandom[3]
datahyp0  =  
        Table[{t, ftrue[t] + Random[Real, {-2, 2}]}, {t, 1, 
            200}];

f[t_]  = Fit[datahyp0, {1, t}, t]
Regress[datahyp0, {1, t}, t, 
      RegressionReport -> {RSquared, 
          EstimatedVariance}]

dataplot  =  
        ListPlot[datahyp0, 
          DisplayFunction -> Identity];
modelplot  =  
        Plot[f[t], {t, 0, 200}, 
          DisplayFunction -> Identity];
Show[dataplot, modelplot, 
      DisplayFunction -> $DisplayFunction]

Notice that we got different fit parameters a and b.

What we can do, is rerun a great many simulations, and see what kind of distribution of the fit parameters a and b we get from the simulation. Here, we shall do 2000 simulations of the experiment, creating 2000 data sets, and get 2000 estimates of the fit parameters.

numdatasets  =  2000;
alist  =  {};
blist  =  {};

Do[
{datahyp0  =  
          Table[{t, ftrue[t] + Random[Real, {-2, 2}]}, {t, 1, 
              200}], 

f[t_]  = 
          Fit[datahyp0, {1, t}, 
            t], 

alist  =  
          Append[alist, Part[f[t], 1]], 
blist  =  
          Append[blist, Part[f[t], 2, 1]]}, {i, 1, numdatasets}]

The best way to present the information that we have collected for the parameters is in a histogram.

Histogram[alist, HistogramCategories -> 20, Frame -> True, 
      FrameLabel -> {"Parameter a",  "Number of \
Occurances"}]
Histogram[blist, HistogramCategories -> 20, Frame -> True, 
      FrameLabel -> {"Parameter b",  "Number of \
Occurances"}]

Mathematica can easily calculate the mean and standard deviation in our parameters:

Mean[alist]
StandardDeviation[alist]

Mean[blist]
StandardDeviation[blist]

So our simulation of the experiment has allowed us to estimate Mother Nature's True Parameters as

a = 3.45 ± 0.16
b = 0.0670 ±0.0014

Note that these numbers may be slightly different if we rerun the code, since the random numbers will be different resulting in different distributions. Of course, if we do a large number of simulations the numbers should not be very different at the end--that is the important point! There is large amount of random error in the original experiment, so we shouldn't expect that we can get much better than this.

Using the Original Data Set to Create New Data Sets

In what we did above, we knew something about the process that was creating the data set that was measured. But what if we don't know the process that created the original data set? Then we cannot reasonably expect to create simulated data sets out of thin air. A lovely idea that has gained acceptance is the bootstrap method, in which simulated data sets are created from the original data set. The name "bootstrap" comes from, I assume, the phrase "pick yourself up by your own bootstraps".

What a weird idea. The key property that your original data set must have for this to work is that the original data set consists of Nindependent and identically distributed data points.

Here is the idea used to create a new, simulated data set, from a small data set so we can see what is happening. The simplest method of bootstrapping is to to pick randomly from the original data set until you have a new data set which is the same size as the original. This will result in some points being duplicated, and some being left out entirely.

data  =  Table[{i, i}, {i, 1, 20}];
datahyp  =  {};

SeedRandom[3]
Do[
datahyp  =  
        Append[datahyp, Part[data, Random[Integer, {1, 20}]]], {i, 1, 
        20}]

TableForm[Sort[datahyp]]

We see in the above table that five data points have been duplicated from the data set, and seven data points deleted.

Let's try this method on our larger data set, and see what we get. The original data set is data0.

datahyp0  =  {};

Do[
datahyp0  =  
        Append[datahyp0, Part[data0, Random[Integer, {1, 200}]]], {i, 1, 
        200}]

f[t_]  = Fit[datahyp0, {1, t}, t]
Regress[datahyp0, {1, t}, t, 
      RegressionReport -> {RSquared, 
          EstimatedVariance}]

dataplot  =  
        ListPlot[datahyp0, 
          DisplayFunction -> Identity];
modelplot  =  
        Plot[f[t], {t, 0, 200}, 
          DisplayFunction -> Identity];
Show[dataplot, modelplot, 
      DisplayFunction -> $DisplayFunction]

Well, we got something different. Let's try doing 2000 simulations, and seeing what we get for a and b.

numdatasets  =  2000;
alist  =  {};
blist  =  {};

Do[
{
datahyp0  =  {}, 
Do[
datahyp0  =  
            Append[datahyp0, Part[data0, Random[Integer, {1, 200}]]], {i, 1, 
            200}], 

f[t_]  = 
          Fit[datahyp0, {1, t}, 
            t], 

alist  =  
          Append[alist, Part[f[t], 1]], 
blist  =  
          Append[blist, Part[f[t], 2, 1]]}, {i, 1, numdatasets}]

The best way to present the information that we have collected for the parameters is in a histogram.

Histogram[alist, HistogramCategories -> 20, Frame -> True, 
      FrameLabel -> {"Parameter a",  "Number of \
Occurances"}]
Histogram[blist, HistogramCategories -> 20, Frame -> True, 
      FrameLabel -> {"Parameter b",  "Number of \
Occurances"}]

Mathematica can easily calculate the mean and standard deviation in our parameters:

Mean[alist]
StandardDeviation[alist]

Mean[blist]
StandardDeviation[blist]

So our simulation of the experiment has allowed us to estimate Mother Nature's True Parameters as

a = 3.59 ± 0.17
b = 0.0662 ±0.0014

These numbers agree, within the error we calculated, with the numbers we found using simulated data sets that were not created from the original distribution. Obviously, the previous method is preferred when you know the process that created the data set. But our little experiement seems to indicate that it really isn't such a crazy idea to create new data sets from the best indicator of the underlying process that created the original data set--the data set itself!

References

The BootStrap Method (excellent discussion which appeared in the New York Times):

http://www.mste.uiuc.edu/stat/bootarticle.html

W.H.Press,S.A.Teukolsky,W.T.Vetterling\&B.P.Flannery,Numerical Recipes in Fortran 77,2nd ed,Cambridge University Press (New York) 1992.