Blog

Using date and time variables

Posted by Michael Schmidt

6/28/13 4:01 PM

This post describes the best way to convert date or time values into numeric time values that can be used in Eureqa.

Time Values in Eureqa:
Eureqa can only store date and time values as numeric values (e.g. total seconds or total days). Therefore, you need to pick a reference point to measure a time duration from, and units to measure the time duration.

For example, you could convert a time value "8:31 am" to 8.52 total hours since midnight. Similarly for dates, you could convert a date like "Dec. 6, 1981 8:31 am" to 81.9 total years since 1900.

You need to make date and time conversions to numeric duration values in another program like Excel before entering into Eureqa (see below for example).

Pitfalls:

1) Do not concatenate date and time strings to get a numeric value. For example, do not convert a date like "1981-12-06" to 19811206. This representation of time is extremely nonlinear. It can preserve order, but has lost all meaning. Additionally, the values are very large and numerically unstable.

2) Avoid measuring time durations from a very distant reference point. For example, if you're data uses time values that span a few days, do not convert these time values to total seconds since the beginning of the century. The numeric values would be enormous and numerically unstable.

Instead, the best practice is to measure a time duration since the time point in your data set.

Convert in Excel:

Many programs can convert date and time values to numeric time duration values. In Excel, if you subtract two date cells, the result is the fractional number of days between the two dates. You could then convert days to hours or some other unit to get numeric values with reasonable numeric magnitudes. For example:
    =(A0-A$0)*24

and then repeated for all rows, would subtract the first date in cell A0, and multiply the resulting day values into hours.
Another useful function is the YEARFRAC function which converts the difference between a date and a reference date to the fraction of years difference between them. For example:
    =YEARFRAC(A$0, A0)
and repeated for all rows, returns the fractional value of years from cell A0.

See Also:
Read More

Topics: Preparing Data, Advanced Techniques, Eureqa, Techniques, Time Series

Use time-delays or time-lags of a variable in Eureqa

Posted by Michael Schmidt

6/28/13 4:01 PM

A time delay retrieves the value of a variable or expression at a fixed offset in the past, according to the time ordering or index of each data point in the data set. This post describes the time-delay building-blocks available in Eureqa and different modeling techniques with delayed values.

Time Delay Building Blocks:

Eureqa provides the delay(x, c) building block to represent an arbitrary time-delay, where x could be any expression. The expression delay(x, c) returns the value of x at c time units in the past. When used as a building-block, Eureqa can automatically optimize expressions or variables to be delayed and the time-delay amount  c.

The figure above plots an arbitrary variable x and a delayed value delay(x, 1.0), where the values are ordered by some time variable t. The delayed version is equal to x at 1.0 time units into the past.

To use time-delay building-blocks, your data must have some notion of time or ordering. You also need to tell Eureqa which variable in your data represents the time or ordering value:


If you don't specify a time variable, Eureqa will use the row number in the spreadsheet as the time value of each data point.

If a particular delayed time value falls between two points in the data set, the value is linearly interpolated between the two data points using the time value.

Eureqa also provides the delay_var(x, c) building-block which is identical to delay(x, c), except that it only accepts a variable as input. It's provided as a special case of the delay(x, c) building-block to allow you to constrain the types delays used in the solutions. But in the end they are effectively identical.

Control the Fraction of Data Used for History

Notice that the delayed output plotted above does not have values on the left side of the graph for the first few time points. This is because these points request previous values of x that lie before the first point in our data set. Eureqa will automatically ignore these data points when calculating errors.

However, there is a way to control how much of the data set Eureqa is allowed to ignore - or effectively, specify a maximum delay offset. You can limit the fraction of data used for time-delay history values in the Advanced Solutions Options menu:


The default maximum fraction is 50% of the data. If you find that Eureqa is identifying solutions with very large time delays, perhaps just to avoid modeling difficult features in the first half of the data set, you may want reduce this fraction

Additionally, you can control the number of delayed values per variable (including a zero delay of an ordinary variable use) in this dialog.

Fixed Time-delays:

Another way to model a value as a function of its previous values is with fixed delays. You can enter in fixed time-delays, or "lags" of the variable, directly into the Search Relationship option. For example:

    x = f( delay(x, 2.1), delay(x, 5.6) )

This search relationship tells Eureqa to find an equation to model the value of x as a function of it's value at 2.1 and 5.6 time units in the past.

Minimum Time-delays:

You may also want to specify a minimum time-delay offset. If you entered a search relationship such as x = f(x), Eureqa would find a trivial answer f(x) = x. More likely, you wanted to find a model of x, but as a function of x at least some amount of time in the past. The way to do this is to again use a fixed delay, such as:

    x = f(delay(x, 3.21))

Here, 3.21 is the minimum time-delay. Now, if the time-delay building-blocks are enabled, Eureqa can delay this delayed input further if necessary.

Delay Differential Equations:

Another common use for time-delays in for modeling using Delay Differential Equations. Finding delay differential equations is just like searching for ordinary differential equations. For example, entering a search relationship like:

    D(y,t) = f(y)

but also enabling time-delay building blocks. This relationship has a trivial solution however: Eureqa will return the slope formula such as

    f(y) = ( y - delay(y, 0.1) )/0.1

Therefore, you most-likely want to limit the total number of delays per variable to one (which includes the zero delay of the normal variable use). You can set this in the Advanced Solution Settings menu. The default is unlimited.

Implementing Delays Outside of Eureqa

In Matlab, you can implement a time delay using the interp1 function. For example, the expression delay(x, 1.23) would be implemented as:


    interp1(t, x, t - 1.23, 'linear')


Implementing delays in Excel is a littler harder. You need to download an Excel add-on that adds an interpolate function. For example, the package XlXtrFun adds a function "Interpolate" that is just like Matlab's interp1. There are also other guides for Linear Interpolation with Excel.
Read More

Topics: Advanced Techniques, Eureqa, Tutorial, Techniques, Time Series

Modeling a derivative such as velocity or acceleration

Posted by Matt Fleming

6/28/13 4:00 PM

Eureqa can automatically estimate numerical derivatives in order to model the rates of change of variables in your data. Often derivatives are more natural and simpler for modeling certain types of phenomena, particularly in physics. This post discusses the basics of entering derivatives into the Eureqa search relationship.

The Derivative Operator:

Eureqa provides the derivative operator D(x, y, n) where x and y are any arbitrary expressions and n is an integer representing the order of the derivative to take. This operator can be used in the Search Relationship setting. For example, consider the search relationship:

    D(x,t,1) = f(x,t)

This relation tells Eureqa to find a function of x and t that models the first derivative (e.g. a velocity or slope) of x with respect to t. Short-hand for the first derivative is D(x,t). The derivative operator can also appear inside the formula as an input variable, for example:

    D(x,t,2) = f( x, D(x,t) )

This relation tells Eureqa to find a model of the second derivative (e.g. an acceleration or curvature) of x with respect to t, as a function of x and the first derivative of x. In Eureqa, this relation will appear as:


Eureqa displays the derivatives in mathematical format after the relationship text is entered.

Alternatively, you could estimate the numerical derivatives ahead of time using another program, and enter these values as a new variable in the data set rather than using Eureqa's derivative operator.
Starting the Search:

Eureqa will calculate the numerical derivatives that appear in your search relation when you start the search. The following screen will appear after you click start:


Eureqa estimates the numerical derivative using a spline fit to the data. This allows more accurate derivative estimates than other methods in case the data contains noise.

Advanced:

Estimating numerical derivatives accurately is a challenging task when the data is sparse or contains noise. Eureqa's derivative estimation is an improvement over the most basic methods like Newton's difference quotient. However, it does not work well in all cases.

One particular problem with spline curves is their accuracy at the head and tail of the data - these points are "surrounded" by fewer data points and thus have higher estimation error. If you can, you might want to ignore these points entirely using a weight variable. Simply add a new column to your data, and set the weight to 1 for all data points but near zero for the first and last 5 to 10 data points.

It may also be worth the effort to estimate the numerical derivatives outside of Eureqa using more specialized tools. For example, you may want to compute the derivative values in R or using Matlab's spline toolbox, and then paste these into Eureqa as a new column variable.
Read More

Topics: Advanced Techniques, Eureqa, Tutorial, Techniques, Numerical Derivatives

Normalizing data variables in Eureqa

Posted by Michael Schmidt

6/28/13 4:00 PM

While normalizing your data variables (rescaling the numeric values) is completely optional, it can greatly improve the performance of Eureqa, and numerical stability of solutions. This post discusses when and how to normalize variables in your data.

When to Normalize:

Eureqa works best when all variables in your data have small to medium magnitudes, on the order of 1 to 100. For example, if you have any variable that ranges over a million, it would be best to rescale the values to larger units.

Additionally, the magnitudes of the variable should be similar to the mean or offset of the variable. For example, if you have a variable that only varies between 100.0 and 100.5, it would be best to subtract off 100 so that it ranges between 0 and 0.5.

For example, consider the following two variables in some data set:
Notice that both variables look rather flat. You can't see any interesting variation because the variable a has such a large offset. Do variables in your data look like this? Let's try subtracting off an offset of 10,000 from a:
Now, we can see some interesting variation in the variable a, but the variable b still looks flat because the variable still has a large magnitude relative to b. Next, let's try dividing the values of a by 50:
Now we can see the interesting variation in both variables, as they now have the same relative scale and magnitudes. This is ideally how we want our data to look before entering it into Eureqa. When the variables are reasonably scaled, Eureqa is most likely to utilize their variation to build accurate solutions.

How to Normalize a Variable:

First, consider changing the units of the data you enter into Eureqa. Could you measure values in meters instead of centimeters? Could you measure currency in millions-of-dollars instead of dollars? Pick units such that the numeric values have a range of approximately 1 to 100.

Second, consider measuring values from an offset. Could you measure time since the time of your first data point, instead of since the beginning of the year or century?

Third, check over your data; look for outliers. Are there any values that are drastically out of proportion with the rest of the values? If so, consider removing this entire row in your data set or giving it a very low weight.

The general formula for normalizing a variable y is:

    y_normalized = (y - offset)/scale

where offset and scale are the normalization parameters. It's recommended that you pick offset and scale manually, so that the numeric values still have an intuitive meaning. However, if you truly don't care what the numeric values mean, a common approach is to set offset equal to the mean of the variable and scale to the standard deviation of the variable.

It's also recommended that you apply normalization before entering your data into Eureqa. However, you can specify the normalization in the Eureqa Search Relation. For example, consider the search relation:

    y = f( x/1000 )

This tells Eureqa to find a model of y as a function of values of x that are divided by 1000.

Automatic Normalization Checks:

By default, Eureqa will check your data for extreme cases of that variables that need to be normalized. When entering or modifying values in the Eureqa data view, you may encounter a message like this:


Here, Eureqa is telling you that the variable y has a large offset. It has a mean value of about 1000, but it only varies by +/- 1.38. Eureqa suggests subtracting 996 from each y value in your data set, but leaving the scale unchanged.

You can also modify this and specify what values to apply. Pick a scale and offset that makes sense and preserves meaning.

See Also:
Read More

Topics: Preparing Data, Eureqa, Tutorial, Techniques, Normalize Data in Eureqa

Subscribe to Our Blog!

Follow Me

Posts by Topic

see all