WibiData’s approach to predictive modeling and experimentation
A conversation I have too often with vendors goes something like:
- “That confidential thing you told me is interesting, and wouldn’t harm you if revealed; probably quite the contrary.”
- “Well, I guess we could let you mention a small subset of it.”
- “I’m sorry, that’s not enough to make for an interesting post.”
That was the genesis of some tidbits I recently dropped about WibiData and predictive modeling, especially but not only in the area of experimentation. However, Wibi just reversed course and said it would be OK for me to tell more or less the full story, as long as I note that we’re talking about something that’s still in beta test, with all the limitations (to the product and my information alike) that beta implies.
As you may recall:
- WibiData started out with a rich technology stack …
- … but decided to cast itself as an application company …
- … whose first vertical market is retailing,
With that as background, WibiData’s approach to predictive modeling as of its next release will go something like this:
- There is still a strong element of classical modeling by data scientists/statisticians, with the models re-scored in batch, perhaps nightly.
- But of course at least some scoring should be done as real-time as possible, to accommodate fresh data such as:
- User interactions earlier in today’s session.
- Technology for today’s session (device, connection speed, etc.)
- Today’s weather.
- WibiData Express is/incorporates a Scala-based language for modeling and query.
- WibiData believes Express plus a small algorithm library gives better results than more mature modeling libraries.
- There is some confirming evidence of this …
- … but WibiData’s customers have by no means switched over yet to doing the bulk of their modeling in Wibi.
- WibiData will allow line-of-business folks to experiment with augmentations to the base models.
- Supporting technology for predictive experimentation in WibiData will include:
- Automated multi-armed bandit testing (in previous versions even A/B testing has been manual).
- A facility for allowing fairly arbitrary code to be included into otherwise conventional model-scoring algorithms, where conventional scoring models can come:
- Straight from WibiData Express.
- Via PMML (Predictive Modeling Markup Language) generated by other modeling tools.
- An appropriate user interface for the line-of-business folks to do certain kinds of injecting.
Let’s talk more about predictive experimentation. WibiData’s paradigm for that is:
- Models are worked out in the usual way.
- Businesspeople have reasons for tweaking the choices the models would otherwise dictate.
- They enter those tweaks as rules.
- The resulting combination — models plus rules — are executed and hence tested.
If those reasons for tweaking are in the form of hypotheses, then the experiment is a test of those hypotheses. However, WibiData has no provision at this time to automagically incorporate successful tweaks back into the base model.
What might those hypotheses be like? It’s a little tough to say, because I don’t know in fine detail what is already captured in the usual modeling process. WibiData gave me only one real-life example, in which somebody hypothesized that shoppers would be in more of a hurry at some times of day than others, and hence would want more streamlined experiences when they could spare less time. Tests confirmed that was correct.
That said, I did grow up around retailing, and so I’ll add:
- Way back in the 1970s, Wal-Mart figured out that in large college towns, clothing in the football team’s colors was wildly popular. I’d hypothesize such a rule at any vendor selling clothing suitable for being worn in stadiums.
- A news event, blockbuster movie or whatever might trigger a sudden change in/addition to fashion. An alert merchant might guess that before the models pick it up. Even better, she might guess which psychographic groups among her customers were most likely to be paying attention.
- Similarly, if a news event caused a sudden shift in buyers’ optimism/pessimism/fear of disaster, I’d test that a response to that immediately.
Finally, data scientists seem to still be a few years away from neatly solving the problem of multiple shopping personas — are you shopping in your business capacity, or for yourself, or for a gift for somebody else (and what can we infer about that person)? Experimentation could help fill the gap.
Comments
2 Responses to “WibiData’s approach to predictive modeling and experimentation”
Leave a Reply
[…] WibiData has some innovative ideas in predictive experimentation. […]
[…] This is the big area for any kind of “closed loop” predictive modeling story, e.g. in experimentation. […]