{"action":"created","issue":{"url":"https://api.github.com/repos/HubSpot/react-experiments/issues/1","labels_url":"https://api.github.com/repos/HubSpot/react-experiments/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/HubSpot/react-experiments/issues/1/comments","events_url":"https://api.github.com/repos/HubSpot/react-experiments/issues/1/events","html_url":"https://github.com/HubSpot/react-experiments/issues/1","id":102308440,"number":1,"title":"A few ideas / thoughts","user":{"login":"eytan","id":93681,"avatar_url":"https://avatars.githubusercontent.com/u/93681?v=3","gravatar_id":"","url":"https://api.github.com/users/eytan","html_url":"https://github.com/eytan","followers_url":"https://api.github.com/users/eytan/followers","following_url":"https://api.github.com/users/eytan/following{/other_user}","gists_url":"https://api.github.com/users/eytan/gists{/gist_id}","starred_url":"https://api.github.com/users/eytan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eytan/subscriptions","organizations_url":"https://api.github.com/users/eytan/orgs","repos_url":"https://api.github.com/users/eytan/repos","events_url":"https://api.github.com/users/eytan/events{/privacy}","received_events_url":"https://api.github.com/users/eytan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-21T06:19:06Z","updated_at":"2015-08-21T14:21:19Z","closed_at":null,"body":"This is awesome!  React and PlanOut always seemed like they could work nicely together, in that PlanOut randomizes parameter values and React components are parametric (via props).\r\n\r\nA few ideas / comments:\r\n\r\n(1) `Experiment` component: A React component's render() method should not modify state, but `renderExposedVariation()`, which gets called by render affects state. It also seems less than ideal to have to re-evaluate all of the logic in `renderExposedVariation()` every time the component needs to be re-rendered.  Why not just save the component you want to be rendered to state in `getInitialState()` or `componentWillMount()`?\r\n\r\nOne thing to watch out for is that you want to make sure that your logging does not depend on your components successfully rendering -- otherwise you might not catch cases where your experiment is causing bugs, and you'll get non-equivalent users in your different experimental conditions.\r\n\r\n(2) Parameters as props: part of the idea behind PlanOut is to get people thinking about experiments in terms of parameterizations of user experiences rather than variants.  Such a notion seems compatible with the way one generally writes React code, in that components' behaviors or features depend on props.\r\n\r\nYou might do this generically by saying something like:\r\n\r\n```\r\n<Experiment experimentClass={foo}>\r\n  ...\r\n  <Parameterize propName1='param1', propName2='param2'>\r\n    <MyComponent/>\r\n  </Parameterize>\r\n  ...\r\n  <Parameterize propName3='param3'>\r\n    ...\r\n  </Parameterize>\r\n  ...\r\n</Experiment>\r\n```\r\n\r\nWhere, for example, `MyComponent` is a component that includes two props, propName1 and propName2, which are populated with `param1` and `param2` from the experiment.\r\n\r\n(3) Variations: I would suggest not marrying experiments to a single parameter, and instead having a `<Variations>` component that performs a function similar to what the existing Experiment component does, e.g.,\r\n\r\n```\r\n<Experiment experimentClass={foo}>\r\n  <Variations variationParam='show_text'>\r\n    <Variant name='experimental'>\r\n      Example A\r\n    </Variant>\r\n    <Variant name='control'>\r\n      Example B\r\n    </Variant>\r\n    <DefaultVariant>\r\n      Things are broken.\r\n    </DefaultVariant>\r\n  </Variations>\r\n</Experiment>\r\n```\r\n\r\n\r\n(4) Logging: If I understand correctly, you are only logging when there is a matching variant.  This seems problematic...\r\n\r\n(i) In the case of the logging example in the README, it's not clear what the treatment/control contrast is.  Under the given scheme, it seems like you would likely only be logging / measuring effects on users who are not necessarily equivalent because those who hit different endpoints could be very different from one another.\r\n\r\n(ii) Logging conditional on matching a known variant is also problematic when the components are causally downstream of one another.  Let's imagine that you have some encouragement on page X that offered a discount (call this D=1), and we only expressed a variant to be shown when D=1, and then on page Y, the purchase form, we either shown the original price (D=0) or the discounted price (D=1).  Visitors with D=0 are folks who would have visited Y regardless of the discount, while those with D=1 include marginal users who would not have otherwise visited (and may therefore be less likely to convert or spend as much).  This means that even if you were to log all users who visit Y, those with D=0 and D=1 represent non-random populations.  This is only an issue because you didn't exposure log all users who hit X, regardless of whether or not we had a variant to display for them.\r\n\r\n(5) Namespace component: Seems interesting, but doesn't seem practical from a code maintainability / readability perspective.  I also wonder how this would play out as you add and remove experiments.\r\n\r\n(6) Have you done much with Flux?  I wonder how transportable this is to that kind of model.\r\n\r\n\r\nOverall really neat stuff, I am looking forward to seeing how things progress with this!"},"comment":{"url":"https://api.github.com/repos/HubSpot/react-experiments/issues/comments/133440778","html_url":"https://github.com/HubSpot/react-experiments/issues/1#issuecomment-133440778","issue_url":"https://api.github.com/repos/HubSpot/react-experiments/issues/1","id":133440778,"user":{"login":"colbyr","id":478109,"avatar_url":"https://avatars.githubusercontent.com/u/478109?v=3","gravatar_id":"","url":"https://api.github.com/users/colbyr","html_url":"https://github.com/colbyr","followers_url":"https://api.github.com/users/colbyr/followers","following_url":"https://api.github.com/users/colbyr/following{/other_user}","gists_url":"https://api.github.com/users/colbyr/gists{/gist_id}","starred_url":"https://api.github.com/users/colbyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colbyr/subscriptions","organizations_url":"https://api.github.com/users/colbyr/orgs","repos_url":"https://api.github.com/users/colbyr/repos","events_url":"https://api.github.com/users/colbyr/events{/privacy}","received_events_url":"https://api.github.com/users/colbyr/received_events","type":"User","site_admin":false},"created_at":"2015-08-21T14:21:18Z","updated_at":"2015-08-21T14:21:18Z","body":"@eytan ah I gotcha :eyes: \r\nTo be clear, @rawls238 is expert/maintainer here, I only meant to contribute some lifecycle knowledge. I think what you're suggesting is excellent feedback."}}