The Away Mission
approach to Validating a New Model
Your mission: To seek
out and understand a new model. To
boldly go where you have not gone before.
Enter the planet’s orbit and do a scan…
A.
Determine
what the planet (the model) is like and if it in habitable
a.
What is the name of the model?
b.
Is it similar to something I am familiar with?
c.
Search the databanks (any databases you can
access) to find research on it
i.
If possible, use academic sources (over say
Wikipedia or blogs--except this one of course!)
ii.
Send communications to analysts that may have
worked with those models—In your communiques, ask if they are familiar with
them and ask if they would be willing to consult with you when you are on the
planet’s surface (or doing more detailed analysis of the model)
iii.
In your study, you may come across new words and
methodologies you are not familiar with—takes note of these and take advantage
of the time to learn
1.
If its helpful, create a flow chart as you study
the model, for example…
d.
Know what you are dealing with prior to beaming
to the surface and pack accordingly (make sure you pack a phaser!)
i.
What assumptions does the model have?
ii.
Are there conditions that are built in?
iii.
In what situations is the model specifically
used for?
iv.
Obtain any formulas that are associated with the
model (i.e. structures seen through the database research)
After beaming down…
B.
Break out your tricorder and do more thorough
scans & tests
a.
Look at how the model was used in the context of
what you are validating (and compare to how it is used in the contexts you
studied)—is it used properly in your context?
b.
Were the assumptions of the model taken into
account?
i.
Here, it may be helpful to test data based on
the assumption of the model. For example, if a technique is based on data
being cointegrated, do a cointegrated test
c.
Are the conditions (if any) reasonable based on
the context you are looking at?
d.
Analyze the dependent and independent variables
i.
Do they make sense to use in the context of your
model (and its type)?
e.
Were there tests performed to find the
best-fitting model (i.e. using AIC or BIC for example)? If the best-fitting model was not used, what
was the reasoning behind it?
f.
Were their sensitivity and back-tests performed
on the model? Do the results show any
weakness and is it explained? Do the
sensitivity tests make sense given their respective scenarios?
g.
What are the tests associated with the
model? Do the results say anything about
the model’s weakness or strength?
Make a report to your commanding officer…
C.
This is the most important part, as your report
will be viewed by people who may or may not know the model you studied
a.
Make sure you show that you took the time to
understand the model and its context and can relay if it was used appropriately
i.
Highlight important components
ii.
Note any assumptions of the model and if they
were properly checked
iii.
Present a summary review of the model strength
results and note any strengths and weaknesses of the model—so they are clear to
decision makers that use it
iv.
Note any strengths or weaknesses discovered when
analyzing the back-tests and sensitivity checks
v.
Present results of any tests you ran (i.e. for
assumptions)
vi.
Make a statement about the validity of the
model…its structure, its use being appropriate (or not), and any comments about
how the model can be improved in the future
