From 8bb50c058da85b45c43b31200eb765eee91e8251 Mon Sep 17 00:00:00 2001 From: Jonathan Bernard Date: Tue, 14 Jun 2011 15:24:57 -0500 Subject: [PATCH] Changed Data Model ------------------ * Created the extended data table. This is a more generic version of the extended data field that was on ``ts_user``. Instead of arbitrary key-value pairs going in a list on the user record we will have an additional table, a one to many relationship between existing tables and the ``ts_ext_data`` table, each row being an extended value of the corresponding record. * Added support to the ``timestamper:create_tables/1`` and ``timestamper_dev:create_table/1`` functions for the new ``ts_ext_data`` table. Documentation ------------- * Added some general docs about the DB layer code. * Added two new issues, *D0020*: Entry exclusion filters and *D0021*: notes width. API Changes ----------- Necessitated by the change to the data model. * Updated ``ts_api`` data-retrieval functions to use extended data: * ``get_user_summary/2`` * ``list_timelines/2`` * ``list_entries/3`` * Updated ``ts_api:put_timeline/3`` to parse the extended data supplied by the caller. *FIXME* it is not actually saving this data. * TODO: Started updating ``ts_api:post_entry/3`` to handle extended data, need to finish. Database Layer -------------- Changes necessitated by the change to the model. * Added ``ts_common:do_set_ext_data/2`` which iterates through the extended data key-value pairs calling ``ts_ext_data:set_property/3``. It does not provide a transaction context. * Split ``ts_common:new/1`` and ``ts_common:update/1`` functions into multiple functions to prevent code-duplication: * ``do_{new,update}`` contains the code that performs integrity checks and the actual database write function. It does this assuming that it is being called from within the context of an mnesia transaction (uses ``mnesia:read`` and ``mnesia:write``). * ``{new,update}/1`` performs the same function as previously. The implementation changed from using mnesia ``dirty_*`` calls to prociding a transaction and calling ``do_{new,update}/1``. * ``{new,update}/2`` expect the record to update/create and the extended data to write atomically with the record. They provide a transaction context, call ``do_{new,update}/1``, then call ``do_set_ext_data/2``. * Similar to the refactoring of ``ts_common:{new,update}/1``, ``ts_entry:{new,update}/1`` have been refactored into multiple methods each to support extended data properties: * ``do_{new,update}/1`` perform the actual update assuming we have already established an mnesia transaction. * ``{new,update}/1`` behave the same as they used to, but now do so by creating an mnesia transaction and calling ``do_{new,update}/1``. * ``{new,update}/2`` create an mnesia transaction, call ``do_{new,update}/1``, and then call ``ts_common:do_set_ext_data/2``. * Again similar to the refactoring of ``ts_common:{new,update}/1``, ``ts_user:{new,update}/1`` have been refactored into multiple methods each to support extended data properties: * ``do_{new,update}/1`` perform the actual update assuming we have already established an mnesia transaction. * ``{new,update}/1`` behave the same as they used to, but now do so by creating an mnesia transaction and calling ``do_{new,update}/1``. * ``{new,update}/2`` create an mnesia transaction, call ``do_{new,update}/1``, and then call ``ts_common:do_set_ext_data/2``. * Created the ``ts_ext_data`` module as the interface to the extended data properties introduced in the data model: * ``create_table/1`` performs the same function as it does in the other db layer modules, creates the table with the appropriate structure given the more general table options desired (location, storage type, etc.). * ``set_property/3`` takes a record, a property key, and a property value as input and sets the property described by the property key on the record to the given value, assuming this is a valid property for the record to have. For example, currently the ``ts_user`` record can have an associated property, ``last_timeline``, which represents the last timeline the user was working with. Trying to pass this property with a ``ts_entry`` record would result in an exception. This function uses ``ts_ext_data:do_set_property/3`` as its underlying implementation. * ``get_property/2`` takes a record and a property key and returns the value of that property for the given record, or ``not_set`` if the property has not been set on that record. This method creates its own mnesia transaction. * ``get_properties/1`` takes a record and returns a list of key-value tuples representing all of the extended data properties set for the given record. This method creates its own mnesia transaction. * ``do_set_property/3`` takes a record reference (not the whole record), a a property key, and adds the property assignment to the ``ts_ext_data`` table. It creates its own mnesia transaction. * Added ``new/2`` and ``update/2`` to the ``ts_timeline`` module to support extended data properties. They delegate implementation to ``ts_common:{new,update}/2``. JSON Encoding/Decoding ---------------------- Changes necessitated by the change to the data model. The JSON objects now contain a potentially unlimited number of fields, as each extended data property is encoded as a seperate field, and looks no different from any of the required fields on the object. The intended explanation in API documentation is that each object type (``user``, ``timeline``, or ``entry``) now has both *required* fields that *MUST* be present in every message in either direction and *optional* fields that may or may not be present in any communication with the API. There should be a clear distinction between which fields are required and which are optional. It might also be a good idea to provide a suggested default for optional values when they are not present. * Updated documentation about JSON record structures to reflect the fact that there are now potentially many optional attributes in addition to the required attributes for each record. * ``record_to_ejson/1`` refactored to ``record_to_ejson/2`` which also takes the extended data attributes and appends them as additional attributes to the end of the record structure. * Created ``ext_data_to_ejson/{1,2}`` to provide a mechanism for reformatting extended data properties whose internal representations are not immediately translatable into JSON. Currently only the ``entry_exclusions`` property, which is a list of strings, needs to be treated this way (changing from ``[val, val]`` to ``{array, [val, val]}`` as needed by ``json:encode/1``. ``ext_data_to_ejson/1`` acts as a more user-friendly facade to ``ext_data_to_ejson/2``. * Rewrote ``ejson_to_record/{2,3}`` and ``ejson_to_record_strict/{2,3}`` to handle extended data. They now use a common method, ``construct_record/3`` to create the actual record object and extended data key-value list. ``ejson_to_record_strict/{2,3}`` only differs in that it checks for the presence of each required field of the record after the record is constructed. The three-parameter versions of these functions also take in the intended reference for the constructed record, replacing anything that is in the EJSON body as the record reference (useful when the body does not have the record ids). These methods now return a tuple: ``{Record, ExtData}`` instead of just the record. * Created ``construct_record/3`` takes a record and the EJSON fields from the input object. The third parameter is an accumulator for the extended data properties found when constructing the record. This method works by iterating over the list of input fields. It recognizes any required fields and updates the record being built with the value. Any fields it does not recognize it assumes are extended data properties and adds to its list. When all input fields have been visited it returns the record and list it has constructed. * ``ejson_to_ext_data/{1,2}`` is the inverse of ``ext_data_to_ejson/{1,2}``. *TODO*: this method is not actually being used by the ``ejson_to_record*`` methods. --- .ide/vim-views/ts_common.erl.view | 107 +++++++++++++++++++++++ db/test/DECISION_TAB.LOG | Bin 156 -> 156 bytes db/test/LATEST.LOG | Bin 92 -> 92 bytes db/test/id_counter.DCD | Bin 130 -> 130 bytes db/test/id_counter.DCL | Bin 169 -> 0 bytes db/test/ts_entry.DCD | Bin 2603 -> 3440 bytes db/test/ts_entry.DCL | Bin 685 -> 0 bytes doc/db_layer.rst | 24 ++++++ doc/issues/desktop/0020fn5.rst | 10 +++ doc/issues/desktop/0021tn5.rst | 7 ++ src/timestamper.erl | 1 + src/timestamper_dev.erl | 1 + src/ts_api.erl | 45 +++++++--- src/ts_common.erl | 52 +++++++++-- src/ts_db_records.hrl | 22 ++++- src/ts_entry.erl | 20 ++++- src/ts_ext_data.erl | 47 ++++++++++ src/ts_json.erl | 138 ++++++++++++++++++------------ src/ts_timeline.erl | 14 +-- src/ts_user.erl | 36 ++++++-- 20 files changed, 431 insertions(+), 93 deletions(-) create mode 100644 .ide/vim-views/ts_common.erl.view delete mode 100644 db/test/id_counter.DCL delete mode 100644 db/test/ts_entry.DCL create mode 100644 doc/db_layer.rst create mode 100644 doc/issues/desktop/0020fn5.rst create mode 100644 doc/issues/desktop/0021tn5.rst create mode 100644 src/ts_ext_data.erl diff --git a/.ide/vim-views/ts_common.erl.view b/.ide/vim-views/ts_common.erl.view new file mode 100644 index 0000000..d3ae340 --- /dev/null +++ b/.ide/vim-views/ts_common.erl.view @@ -0,0 +1,107 @@ +let s:so_save = &so | let s:siso_save = &siso | set so=0 siso=0 +argglobal +edit ~/projects/jdb-labs/timestamper/web-app/src/ts_common.erl +setlocal keymap= +setlocal noarabic +setlocal autoindent +setlocal balloonexpr= +setlocal nobinary +setlocal bufhidden= +setlocal buflisted +setlocal buftype= +setlocal nocindent +setlocal cinkeys=0{,0},0),:,0#,!^F,o,O,e +setlocal cinoptions= +setlocal cinwords=if,else,while,do,for,switch +setlocal comments=s1:/*,mb:*,ex:*/,://,b:#,:%,:XCOMM,n:>,fb:- +setlocal commentstring=/*%s*/ +setlocal complete=.,w,b,u,t,i +setlocal completefunc= +setlocal nocopyindent +setlocal nocursorcolumn +setlocal nocursorline +setlocal define= +setlocal dictionary= +setlocal nodiff +setlocal equalprg= +setlocal errorformat= +setlocal expandtab +if &filetype != 'erlang' +setlocal filetype=erlang +endif +setlocal foldcolumn=0 +setlocal foldenable +setlocal foldexpr=0 +setlocal foldignore=# +setlocal foldlevel=0 +setlocal foldmarker={{{,}}} +setlocal foldmethod=manual +setlocal foldminlines=1 +setlocal foldnestmax=20 +setlocal foldtext=foldtext() +setlocal formatexpr= +setlocal formatoptions=tcq +setlocal formatlistpat=^\\s*\\d\\+[\\]:.)}\\t\ ]\\s* +setlocal grepprg= +setlocal iminsert=2 +setlocal imsearch=2 +setlocal include= +setlocal includeexpr= +setlocal indentexpr= +setlocal indentkeys=0{,0},:,0#,!^F,o,O,e +setlocal noinfercase +setlocal iskeyword=@,48-57,_,192-255 +setlocal keywordprg= +setlocal nolinebreak +setlocal nolisp +setlocal nolist +setlocal makeprg= +setlocal matchpairs=(:),{:},[:] +setlocal nomodeline +setlocal modifiable +setlocal nrformats=octal,hex +setlocal number +setlocal numberwidth=4 +setlocal omnifunc= +setlocal path= +setlocal nopreserveindent +setlocal nopreviewwindow +setlocal quoteescape=\\ +setlocal noreadonly +setlocal norightleft +setlocal rightleftcmd=search +setlocal noscrollbind +setlocal shiftwidth=4 +setlocal noshortname +setlocal nosmartindent +setlocal softtabstop=0 +setlocal nospell +setlocal spellcapcheck=[.?!]\\_[\\])'\"\ \ ]\\+ +setlocal spellfile= +setlocal spelllang=en +setlocal statusline= +setlocal suffixesadd= +setlocal swapfile +setlocal synmaxcol=3000 +if &syntax != 'erlang' +setlocal syntax=erlang +endif +setlocal tabstop=4 +setlocal tags= +setlocal textwidth=80 +setlocal thesaurus= +setlocal nowinfixheight +setlocal nowinfixwidth +setlocal wrap +setlocal wrapmargin=0 +silent! normal! zE +let s:l = 16 - ((14 * winheight(0) + 23) / 46) +if s:l < 1 | let s:l = 1 | endif +exe s:l +normal! zt +16 +normal! 034l +let &so = s:so_save | let &siso = s:siso_save +doautoall SessionLoadPost +" vim: set ft=vim : +syntax on diff --git a/db/test/DECISION_TAB.LOG b/db/test/DECISION_TAB.LOG index 076bf02028f8f5074a25d5db92d170645ada506d..c41385d58f11381a7db61f90c6a38d35d7bfc444 100644 GIT binary patch delta 24 fcmbQkIEQgU2)9fU1EWZ95(9IL_r!Q>rsG)vPoW01 delta 24 gcmbQkIEQgU2)A?+1BdJ4BnF0!ZzslEGi}ZS09r~1;Q#;t diff --git a/db/test/LATEST.LOG b/db/test/LATEST.LOG index 694f0dadb58ba5b346f96e246db53654069362d5..761214661f0567a51a86cf6d6a8330757ac7f78c 100644 GIT binary patch delta 16 Xcma!vnGnb=lf=L%(woG<{G$c{B>n`R delta 16 Xcma!vnGnb=oy5T5x+sZ(?^+!ICf@|r diff --git a/db/test/id_counter.DCD b/db/test/id_counter.DCD index 1594640a33f311b650e7a38b26417d04cea10d33..2adfa9428ccf187570404ddd91bdaf6180f8bf71 100644 GIT binary patch delta 23 ecmZo-Y+{_?&n=V0z$nt2#K3xrWnz>%qcs3Tr3I7# delta 23 ecmZo-Y+{_?&n=zAz~S;PiGlBZ)5IusMl%3Ul?KuP diff --git a/db/test/id_counter.DCL b/db/test/id_counter.DCL deleted file mode 100644 index a83bc9f09a3831920dc37fabb8bffc8f22b491f7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 169 zcmZQ%VrEH>@a1M;VDL%`_i=2_U`t`(%E?cU&qz&7Ni9lYU{6WTi3bW~GcX(K8DulC zndq7585*T9@aE;`<)@@N4UaA%WX?#WTGcECMsNy`gSn6PUYM%{#_7rN-7 zFToCkKwCm2Eg?aY7SQ1`5b;=B9e-4cRkB)qPON@&yfmk|MR0P|oh3(CgTJSS7d0}B zj}yb{|H0P{6B>>KII!J*!~-8X+?6OI=mY|&jNlmQJvLAx3(l=!0{W4X$N(&Q7TCoB z{`sL$8-aKIKy0G}+y{Az;P5DPt)UKkKXDOwWJyxmWJPVt3Z>`I39Qrs{JzdAD%eJp44xo@A$yLA8n delta 35 rcmew$wOV9?KfiPm1BdqfBnE~^uhV z&?n$1OIc7muvwPv@Av<2G=gUKaI*sdSLyxr`KxLnY!~)vqDUe+giXwfiEn5KdU2wm z)sOpe?+Br7tg%RE#wuI03Q~aZAcd$mPhoFx=r8J@i-Bq&L|jkEuzOK~hOH^4?#7Iq=A6hTq83yS$_^%d#Rj z5*JC5oQ%X}Srt_J|AZ7MnYvY;X>#JxI#IK9EZA?w%SvbeuJrhCrI~e(GG5Xr2y?bB n*0A5XwZd4Ydhr6QycUb};?9!UYvwy#(d?l&yZyc}`**%ygof71 diff --git a/doc/db_layer.rst b/doc/db_layer.rst new file mode 100644 index 0000000..66e7406 --- /dev/null +++ b/doc/db_layer.rst @@ -0,0 +1,24 @@ +TimeStamper DB Layer +==================== + +The following modules make up the database layer: + +* ``ts_user``: Interface to user data. +* ``ts_timeline``: Interface to timeline data. +* ``ts_entry``: Interface to timeline entry data. +* ``ts_ext_data``: Interface to extended data that can be set on different + records. +* ``ts_db_records``: Definition of data records. + +The following modules and files are implementation details of the DB layer: + +* ``id_counter``: Adds support for unique, sequential ID generation. +* ``ts_common``: Provides the implementation for any operations that are common + between the different interfaces. + +Philosophy +---------- + +The database layer should abstract all database-specific code away from the +caller. Users of the DB layer should not have to think about transactions, +locking, etc. diff --git a/doc/issues/desktop/0020fn5.rst b/doc/issues/desktop/0020fn5.rst new file mode 100644 index 0000000..336b231 --- /dev/null +++ b/doc/issues/desktop/0020fn5.rst @@ -0,0 +1,10 @@ +Add exclusion filter for entries. +================================= + +Add a way for users to add regexes for entries to be ignored in the +display. Exclusions may be per-timeline or per-user. + +========= ========== +Created: 2011-06-10 +Resolved: YYYY-MM-DD +========= ========== \ No newline at end of file diff --git a/doc/issues/desktop/0021tn5.rst b/doc/issues/desktop/0021tn5.rst new file mode 100644 index 0000000..81aaf2f --- /dev/null +++ b/doc/issues/desktop/0021tn5.rst @@ -0,0 +1,7 @@ +Constrain width of notes fields to width of mark. +================================================= + +========= ========== +Created: 2011-06-10 +Resolved: YYYY-MM-DD +==================== \ No newline at end of file diff --git a/src/timestamper.erl b/src/timestamper.erl index 8e115ff..ab677a3 100644 --- a/src/timestamper.erl +++ b/src/timestamper.erl @@ -13,4 +13,5 @@ create_tables(Nodes) -> {atomic, ok} = ts_user:create_table(TableOpts), {atomic, ok} = ts_timeline:create_table(TableOpts), {atomic, ok} = ts_entry:create_table(TableOpts), + {atomic, ok} = ts_ext_data:create_table(TableOpts), ok. diff --git a/src/timestamper_dev.erl b/src/timestamper_dev.erl index 0ba7038..c2043d8 100644 --- a/src/timestamper_dev.erl +++ b/src/timestamper_dev.erl @@ -14,4 +14,5 @@ create_tables(Nodes) -> {atomic, ok} = ts_user:create_table(TableOpts), {atomic, ok} = ts_timeline:create_table(TableOpts), {atomic, ok} = ts_entry:create_table(TableOpts), + {atomic, ok} = ts_ext_data:create_table(TableOpts), ok. diff --git a/src/ts_api.erl b/src/ts_api.erl index e00d20e..fff4c59 100644 --- a/src/ts_api.erl +++ b/src/ts_api.erl @@ -204,11 +204,17 @@ get_user_summary(YArg, Username) -> case ts_user:lookup(Username) of no_record -> make_json_404(YArg); User -> - EJSONUser = ts_json:record_to_ejson(User), + UserExtData - ts_ext_data:get_properties(Username), + EJSONUser = ts_json:record_to_ejson(User, UserExtData), Timelines = ts_timeline:list(Username, 0, 100), EJSONTimelines = {array, - lists:map(fun ts_json:record_to_ejson/1, Timelines)}, + lists:map( + fun(Timeline) -> + ts_json:record_to_ejson(Timeline, + ts_ext_data:get_properties(Timeline#ts_timeline.ref)) + end, + Timelines)}, JSONResp = json:encode({struct, [{user, EJSONUser}, @@ -245,7 +251,12 @@ list_timelines(YArg, Username) -> Timelines = ts_timeline:list(Username, Start, Length), % convert them all to their EJSON form - EJSONTimelines = {array, lists:map(fun ts_json:record_to_ejson/1, Timelines)}, + EJSONTimelines = {array, lists:map( + fun (Timeline) -> + ts_json:record_to_ejson(Timeline, + ts_ext_data:get_properties(Timeline#ts_timeline.ref)) + end, + Timelines)}, % create resposne JSONResponse = json:encode(EJSONTimelines), @@ -268,12 +279,13 @@ put_timeline(YArg, Username, TimelineId) -> %{struct, Fields} = EJSON, % parse into a timeline record - TR = try ts_json:ejson_to_record_strict( + {TR, ExtData} = + try ts_json:ejson_to_record_strict( #ts_timeline{ref={Username, TimelineId}}, EJSON) - catch _:InputError -> + catch throw:{InputError, _StackTrace} -> error_logger:error_report("Bad input: ~p", [InputError]), - throw(make_json_400(YArg)) - end, + throw(make_json_400(YArg, {request_error, InputError)) + end, ts_timeline:write(TR), make_json_200(YArg, TR). @@ -321,7 +333,11 @@ list_entries(YArg, Username, TimelineId) -> end, EJSONEntries = {array, lists:map( - fun ts_json:record_to_ejson/1, Entries)}, + fun (Entry) -> + ts_json:record_to_ejson(Entry, + ts_ext_data:get_properties(Entry#ts_entry.ref)), + end, + Entries)}, JSONResponse = json:encode(EJSONEntries), @@ -353,7 +369,11 @@ list_entries(YArg, Username, TimelineId) -> end, EJSONEntries = {array, lists:map( - fun ts_json:record_to_ejson/1, Entries)}, + fun (Entry) -> + ts_json:record_to_ejson(Entry, + ts_ext_data:get_properties(Entry#ts_entry.ref)), + end, + Entries)}, JSONResponse = json:encode(EJSONEntries), @@ -374,16 +394,18 @@ post_entry(YArg, Username, TimelineId) -> EJSON = parse_json_body(YArg), % parse into ts_entry record - ER = try ts_json:ejson_to_record_strict( + {ER, ExtData} = try ts_json:ejson_to_record_strict( #ts_entry{ref = {Username, TimelineId, undefined}}, EJSON) catch _:InputError -> error_logger:error_report("Bad input: ~p", [InputError]), throw(make_json_400(YArg)) end, + %% TODO; should entries and their properties be created atomically? case ts_entry:new(ER) of % record created {ok, CreatedRecord} -> + [{status, 201}, make_json_200(YArg, CreatedRecord)]; % will not create, record exists @@ -445,7 +467,8 @@ parse_json_body(YArg) -> %% Create a JSON 200 response. make_json_200(_YArg, Record) -> - JSONResponse = json:encode(ts_json:record_to_ejson(Record)), + RecordExtData = ts_ext_data:get_properties(element(2, Record)), + JSONResponse = json:encode(ts_json:record_to_ejson(Record, RecordExtData)), {content, "application/json", JSONResponse}. make_json_400(YArg) -> make_json_400(YArg, []). diff --git a/src/ts_common.erl b/src/ts_common.erl index ee33efd..c39f40c 100644 --- a/src/ts_common.erl +++ b/src/ts_common.erl @@ -1,32 +1,64 @@ -module(ts_common). --export([new/1, update/1, update_record/2, list/3, order_datetimes/2]). +-export([new/1, new/2, update/1, update/2, update_record/2, do_set_ext_data/2, + list/3, order_datetimes/2]). -include_lib("stdlib/include/qlc.hrl"). new(Record) -> + {atomic, Result} = mnesia:transaction(fun() -> do_new(Record) end), + Result. +new(Record, ExtData) when is_list(ExtData) -> + {atomic, Result} = mnesia:transaction(fun() -> + case do_new(Record) of + {error, Err} -> mnesia:abort({"Cannot create new record.", Err}); + NewRecord -> case do_set_ext_data(Record, ExtData) of + ok -> NewRecord; + Error -> mnesia:abort(Error) + end + end + end), + Result. + +% required to be wrapped in a transaction +do_new(Record) -> Table = element(1, Record), % check for existing record - case mnesia:dirty_read(Table, element(2, Record)) of + case mnesia:read(Table, element(2, Record)) of % record exists [ExistingRecord] -> {error, {record_exists, ExistingRecord}}; - [] -> mnesia:dirty_write(Record) + [] -> mnesia:write(Record) end. update(Record) -> + {atomic, Result} = mnesia:transaction(fun() -> do_update(Record) end), + Result. +update(Record, ExtData) when is_list(ExtData) -> + {atomic, Result} = mnesia:transaction(fun() -> + case do_update(Record) of + {error, Err} -> mnesia:abort({"Cannot update record.", Err}); + UpdatedRecord -> case do_set_ext_data(Record, ExtData) of + of -> UpdatedRecord; + Error -> mnesia:abort({"Cannot update record.", Error}) + end + end + end), + Result. + +do_update(Record) -> Table = element(1, Record), Key = element(2, Record), % look for existing record - case mnesia:dirty_read(Table, Key) of + case mnesia:read(Table, Key) of % record does not exist, cannot update [] -> no_record; % record does exist, update [ExistingRecord] -> - mnesia:dirty_write(update_record(ExistingRecord, Record)) + mnesia:write(update_record(ExistingRecord, Record)) end. update_record(Record, UpdateData) -> @@ -66,6 +98,16 @@ list(Query, Start, Length) -> end), Result. +% should be wrapped in a transaction +do_set_ext_data(Record, ExtData) when is_list(ExtData) -> + Ref = element(2, Record), + lists:foreach( + fun({Key, Val}) -> + {atomic, ok} = ts_ext_data:set_property(Ref, Key, Val) + end, + ExtData) + ok. + % This is somewhat ridiculous. order_datetimes({{Y1, Mon1, D1}, {H1, Min1, S1}}, {{Y2, Mon2, D2}, {H2, Min2, S2}}) -> diff --git a/src/ts_db_records.hrl b/src/ts_db_records.hrl index 2d94a7b..cab0f00 100644 --- a/src/ts_db_records.hrl +++ b/src/ts_db_records.hrl @@ -4,12 +4,21 @@ pwd_salt, name, email, - join_date, - ext_data = [] % other extensible data + join_date }). -% ts_user.ext_data can be: -% [{last_timeline, TimelineId}] +% ts_user.ext_data format is +% [{key, val}, {key, val}] +% +% Valid key/value pairs are: +% +% * last_timeline/TimelineId +% +% list() representing a timeline owned by this user +% +% * entry_exclusions/EntryExclusions +% +% list() of regular expressions to be excluded from display, seperated by '|' -record(ts_timeline, { ref, % {username, timelineid} @@ -29,6 +38,11 @@ expires }). +% extensible data for arbitrary entities +-record(ts_ext_data, { + ref, % {ref, key}: reference with item ref and property key + value % value +}). %-record(ts_session, { %session_id, %expires, diff --git a/src/ts_entry.erl b/src/ts_entry.erl index 13eeed8..3662f08 100644 --- a/src/ts_entry.erl +++ b/src/ts_entry.erl @@ -1,5 +1,6 @@ -module(ts_entry). --export([create_table/1, new/1, update/1, write/1, delete/1, lookup/3, list_asc/3, list_desc/3]). +-export([create_table/1, new/1, new/2, update/1, update/2, write/1, delete/1, + lookup/3, list_asc/3, list_desc/3]). -include("ts_db_records.hrl"). -include_lib("stdlib/include/qlc.hrl"). @@ -10,15 +11,27 @@ create_table(TableOpts) -> {type, ordered_set}, {index, [timestamp]}]). new(ER = #ts_entry{}) -> + {atomic, NewRow} = mnesia:transaction(fun() -> do_new(ER) end), + {ok, NewRow}. + +new(ER = #ts_entry{}, ExtData) when is_list(ExtData) -> {atomic, NewRow} = mnesia:transaction(fun() -> + NewRow = do_new(ER), + ts_common:do_set_ext_data(ER, ExtData), + NewRow end), + NewRow. + +do_new(ER = #ts_entry{}) -> {Username, TimelineId, _} = ER#ts_entry.ref, NextId = id_counter:next_counter(ts_entry_id), NewRow = ER#ts_entry{ref = {Username, TimelineId, NextId}}, ok = mnesia:write(NewRow), - NewRow end), - {ok, NewRow}. + NewRow. update(ER = #ts_entry{}) -> ts_common:update(ER). + +update(ER = #ts_entry{}, ExtData) when is_list(ExtData) -> + ts_common:update(ER, ExtData). write(ER = #ts_entry{}) -> mnesia:dirty_write(ER). @@ -30,7 +43,6 @@ lookup(Username, TimelineId, EntryId) -> delete(ER = #ts_entry{}) -> mnesia:dirty_delete_object(ER). - list({Username, Timeline}, Start, Length, OrderFun) when is_integer(Start) and is_integer(Length) -> diff --git a/src/ts_ext_data.erl b/src/ts_ext_data.erl new file mode 100644 index 0000000..384a3b8 --- /dev/null +++ b/src/ts_ext_data.erl @@ -0,0 +1,47 @@ +-module(ts_ext_data). +-export([create_table/1, set_property/3, get_property/2, get_properties/1]). + +-include("ts_db_records.hrl"). + +create_table(TableOpts) -> + mnesia:create_table(ts_entry, + TableOpts ++ [{attributes, record_info(fields, ts_ext_data)}, + {type, ordered_set}]). + +% set last timeline +set_property(Ref=#ts_user{}, last_timeline, LastTimelineId) -> + do_set_property(Ref, last_timeline, LastTimelineId); + +% Set exclusion_list for a User account +set_property(Ref=#ts_user{}, entry_exclusions, ExclusionList) -> + do_set_property(Ref, entry_exclusions, string:join(ExclusionList, "|")); + +% Set exclusion_list for a Timeline entry +set_property(Ref=#ts_timeline{}, entry_exclusions, ExclusionList) -> + do_set_property(Ref, entry_exclusions, string:join(ExclusionList, "|")); + +set_property(Ref, Key, Value) -> + throw(io_lib:format("Property '~s' not available for a ~s record.", + [Key, element(1, Ref)]). + +get_property(Ref, PropKey) -> + {atomic, Result} = mnesia:transaction(fun() -> + case mnesia:read(ts_ext_data, {Ref, PropKey}) of + [] -> not_set; + [Property] -> Property + end + end), + Result. + +get_properties(Ref) -> + {atomic, Result} = mnesia:transaction(fun() -> + MatchHead = #ts_ext_data{ref = {Ref, '_'}, _='_'}, + mnesia:select(ts_ext_data, [{MatchHead, [], ['$_']}]) + end), + Result. + +do_set_property(Ref, PropKey, Val) -> + {atomic, Result} = mnesia:transaction(fun() -> + mnesia:write(#ts_ext_data{ref = {Ref, PropKey}, value = Val}). + end), + Result. diff --git a/src/ts_json.erl b/src/ts_json.erl index 9a6a942..55ec0f5 100644 --- a/src/ts_json.erl +++ b/src/ts_json.erl @@ -1,24 +1,25 @@ -module(ts_json). --export([encode_record/1, record_to_ejson/1, ejson_to_record/2, ejson_to_record_strict/2]). +-export([encode_record/1, record_to_ejson/2, + ejson_to_record/2, ejson_to_record/3, + ejson_to_record_strict/2, ejson_to_record_strict/3]). -include("ts_db_records.hrl"). encode_record(Record) -> lists:flatten(json:encode(record_to_ejson(Record))). -% User JSON record structure: +% User JSON record required structure: % {"id": "john_doe", % "name": "John Doe", % "email": "john.doe@example.com", -% "join_date": "2011-01-01T12:00.000Z", -% "ext_data": {"last_timeline", "personal"}} +% "join_date": "2011-01-01T12:00.000Z"} -record_to_ejson(Record=#ts_user{}) -> +record_to_ejson(Record=#ts_user{}, ExtData) -> {struct, [ {id, Record#ts_user.username}, {name, Record#ts_user.name}, {email, Record#ts_user.email}, - {join_date, encode_datetime(Record#ts_user.join_date)}, - {ext_data, Record#ts_user.ext_data}]}; + {join_date, encode_datetime(Record#ts_user.join_date)}] ++ + ext_data_to_ejson(ExtData)}; % Timeline JSON record stucture: % {"user_id": "john_doe", @@ -26,7 +27,7 @@ record_to_ejson(Record=#ts_user{}) -> % "created": "2011-01-01T14:00.000Z", % "description:"Personal time-tracking."} -record_to_ejson(Record=#ts_timeline{}) -> +record_to_ejson(Record=#ts_timeline{}, ExtData) -> % pull out the username and timeline id {Username, TimelineId} = Record#ts_timeline.ref, @@ -35,7 +36,8 @@ record_to_ejson(Record=#ts_timeline{}) -> {user_id, Username}, {id, TimelineId}, {created, encode_datetime(Record#ts_timeline.created)}, - {description, Record#ts_timeline.desc}]}; + {description, Record#ts_timeline.desc}] ++ + ext_data_to_ejson(ExtData)}; % Entry JSON record structure: % {"user_id": "john_doe", @@ -45,7 +47,7 @@ record_to_ejson(Record=#ts_timeline{}) -> % "mark": "Workout.", % "notes": "First workout after a long break."} -record_to_ejson(Record=#ts_entry{}) -> +record_to_ejson(Record=#ts_entry{}, ExtData) -> % pull out the username, timeline id, and entry id {Username, TimelineId, EntryId} = Record#ts_entry.ref, @@ -59,68 +61,78 @@ record_to_ejson(Record=#ts_entry{}) -> {id, EntryId}, {timestamp, encode_datetime(DateTime)}, {mark, Record#ts_entry.mark}, - {notes, Record#ts_entry.notes}]}. + {notes, Record#ts_entry.notes}] ++ + ext_data_to_ejson(ExtData)}. encode_datetime({{Year, Month, Day}, {Hour, Minute, Second}}) -> lists:flatten(io_lib:format("~4.10.0B-~2.10.0B-~2.10.0BT~2.10.0B:~2.10.0B:~2.10.0B.~3.10.0BZ", [Year, Month, Day, Hour, Minute, Second, 000])). -ejson_to_record(Rec=#ts_timeline{}, EJSON) -> - {struct, Fields} = EJSON, +ext_data_to_ejson(ExtData) -> ext_data_to_ejson(ExtData, []). - Created = case lists:keyfind(created, 1, Fields) of - false -> undefined; - {created, CreatedVal} -> decode_datetime(CreatedVal) - end, +ext_data_to_ejson([], ExtData) -> ExtData; - Desc = case lists:keyfind(description, 1, Fields) of - false -> undefined; - {description, DescVal} -> DescVal - end, +ext_data_to_ejson([{Key, Value}|T], Acc) -> + case Key of + entry_exclusions -> ext_data_to_ejson(T, [{Key, {array, Value}}|Acc]); + _Other -> ext_data_to_json(T, [{Key, Value}|Acc]) + end. - Rec#ts_timeline{ - created = Created, - desc = Desc }; +ejson_to_record(Empty, {struct, EJSONFields}) -> + construct_record(Empty, EJSONFields, []). -ejson_to_record(Rec=#ts_entry{}, EJSON) -> - {struct, Fields} = EJSON, +ejson_to_record(Empty, Ref, EJSON) -> + Constructed = ejson_to_record(Empty, EJSON), + setelement(2, Constructed, Ref). - Timestamp = case lists:keyfind(timestamp, 1, Fields) of - false -> undefined; - {timestamp, TSVal} -> calendar:datetime_to_gregorian_seconds( - decode_datetime(TSVal)) - end, +ejson_to_record_strict(Empty=#ts_timeline{}, EJSON) -> + Constructed = ejson_to_record(Empty, EJSON), + case Constructed of + #ts_timeline{created = undefined} -> + throw("Missing timeline 'created' field."); + #ts_timeline{desc = undefined} -> + throw("Missing timeline 'description' field."); + _Other -> Constructed + end; - Mark = case lists:keyfind(mark, 1, Fields) of - false -> undefined; - {mark, MarkVal} -> MarkVal - end, +ejson_to_record_strict(Empty=#ts_entry{}, EJSON) -> + Constructed = ejson_to_record(Empty, EJSON), + case Constructed of + #ts_entry{timestamp = undefined} -> + throw("Missing timelne 'timestamp' field."); + #ts_entry{mark = undefined} -> + throw("Missing timeline 'mark' field."); + #ts_entry{notes = undefined} -> + throw("Missing timeline 'notes' field/"); + _Other -> Constructed + end. - Notes = case lists:keyfind(notes, 1, Fields) of - false -> undefined; - {notes, NotesVal} -> NotesVal - end, +ejson_to_record_strict(Empty, Ref, EJSON) -> + Constructed = ejson_to_record_strict(Empty, EJSON), + setelement(2, Constructed, Ref). - Rec#ts_entry{ - timestamp = Timestamp, - mark = Mark, - notes = Notes}. +construct_record(Timeline=#ts_timeline{}, [{Key, Val}|Fields], ExtData) -> + case Key of + created -> construct_record( + Timeline#ts_timeline{created = decode_datetime(Value)}, + Fields, ExtData); + description -> construct_record(Timeline#ts_timeline{desc = Value}, + Fields, ExtData); + Other -> construct_record(Timeline, Fields, [{Key, Value}|ExtData]) + end; -ejson_to_record_strict(Rec=#ts_timeline{}, EJSON) -> - {struct, Fields} = EJSON, +construct_record(Entry=#ts_entry{}, [{Key, Value}|Fields], ExtData) -> + case Key of + timestamp -> construct_record( + Entry#ts_entry{timestamp = calendar:datetime_to_gregoraian_seconds( + decode_datetime(Value))}, + Fields, ExtData); + mark -> construct_record(Entry#ts_entry{mark=Value}, Fields, ExtData); + notes -> construct_record(Entry#ts_entry{notes=Vale}, Fields, ExtData); + Other -> construct_record(Entry, Fields, [{Key, Value}|ExtData]) + end; - Rec#ts_timeline{ - created = decode_datetime(element(2, lists:keyfind(created, 1, Fields))), - desc = element(2, lists:keyfind(description, 1, Fields))}; - -ejson_to_record_strict(Rec=#ts_entry{}, EJSON) -> - {struct, Fields} = EJSON, - - Rec#ts_entry{ - timestamp = calendar:datetime_to_gregorian_seconds(decode_datetime( - element(2, lists:keyfind(timestamp, 1, Fields)))), - mark = element(2, lists:keyfind(mark, 1, Fields)), - notes = element(2, lists:keyfind(notes, 1, Fields))}. +construct_record(Record, [], ExtData) -> {Record, ExtData}. decode_datetime(DateTimeString) -> % TODO: catch badmatch and badarg on whole function @@ -144,3 +156,15 @@ decode_datetime(DateTimeString) -> list_to_integer(SecondString)}, {Date, Time}. + +ejson_to_ext_data(ExtData) -> ejson_to_ext_data(ExtData, []). + +ejson_to_ext_data([], ExtData) -> ExtData; + +ejson_to_ext_data([{Key, Value}|T], Acc) -> + case Key of + entry_exclusions -> ext_data_to_ejson(T, [{Key, element(2, Value)}|Acc]); + _Other -> ext_data_to_json(T, [{Key, Value}|Acc]) + end. + + diff --git a/src/ts_timeline.erl b/src/ts_timeline.erl index 547c4d0..e91066e 100644 --- a/src/ts_timeline.erl +++ b/src/ts_timeline.erl @@ -1,5 +1,5 @@ -module(ts_timeline). --export([create_table/1, new/1, update/1, write/1, lookup/2, list/3]). +-export([create_table/1, new/1, new/2, update/1, update/2, write/1, lookup/2, list/3]). -include("ts_db_records.hrl"). -include_lib("stdlib/include/qlc.hrl"). @@ -11,8 +11,14 @@ create_table(TableOpts) -> new(TR = #ts_timeline{}) -> ts_common:new(TR). +new(TR = #ts_timeline{}, ExtData) when is_list(ExtData) -> + ts_common:new(TR, ExtData). + update(TR = #ts_timeline{}) -> ts_common:update(TR). +update(TR = #ts_timeline{}, ExtData) when is_list(ExtData) -> + ts_common:update(TR, ExtData). + write(TR = #ts_timeline{}) -> mnesia:dirty_write(TR). lookup(Username, TimelineId) -> @@ -22,8 +28,6 @@ lookup(Username, TimelineId) -> end. list(Username, Start, Length) -> - {atomic, Timelines} = mnesia:transaction(fun() -> - MatchHead = #ts_timeline{ref = {Username, '_'}, _='_'}, - mnesia:select(ts_timeline, [{MatchHead, [], ['$_']}]) - end), + MatchHead = #ts_timeline{ref = {Username, '_'}, _='_'}, + mnesia:dirty_select(ts_timeline, [{MatchHead, [], ['$_']}]), lists:sublist(Timelines, Start + 1, Length). diff --git a/src/ts_user.erl b/src/ts_user.erl index b95019c..2596908 100644 --- a/src/ts_user.erl +++ b/src/ts_user.erl @@ -1,6 +1,6 @@ -module(ts_user). -%-export([create_table/1, new/1, update/1, lookup/1, list/2]). --compile(export_all). +-export([create_table/1, new/1, new/2, update/1, update/2, lookup/1, list/2, + check_credentials/2]). -include("ts_db_records.hrl"). -include_lib("stdlib/include/qlc.hrl"). @@ -12,13 +12,35 @@ create_table(TableOpts) -> % expects the password in clear new(UR = #ts_user{}) -> - case mnesia:dirty_read(ts_user, UR#ts_user.username) of + {atomic, Result} = mnesia:transaction(fun() -> do_new(UR) end), + Result. + +new(UR = #ts_user{}, ExtData) -> + {atomic, Result} = mnesia:transaction(fun() -> + NewUser = do_new(UR), + ts_common:do_set_ext_data(UR, ExtData), + NewUser end), + Result. + +do_new(UR = #ts_user{}) -> + case mnesia:read(ts_user, UR#ts_user.username) of [ExistingRecord] -> {error, {record_exists, ExistingRecord}}; - [] -> mnesia:dirty_write(hash_input_record(UR)) + [] -> mnesia:write(hash_input_record(UR)) end. update(UR = #ts_user{}) -> - case mnesia:dirty_read(ts_user, UR#ts_user.username) of + {atomic, Result} = mnesia:transaction(fun() -> do_update(UR) end), + Result. + +update(UR = #ts_user{}, ExtData) -> + {atomic, Result} = mnesia:transaction(fun() -> + UpdatedUser = do_update(UR), + ts_common:do_set_ext_data(UR, ExtData) + UpdatedUser end), + Result. + +do_update(UR = #ts_user{}) -> + case mnesia:read(ts_user, UR#ts_user.username) of [] -> no_record; [Record] -> UpdatedRecord = ts_common:update_record(Record, UR), @@ -26,14 +48,14 @@ update(UR = #ts_user{}) -> undefined -> UpdatedRecord; _Password -> hash_input_record(UpdatedRecord) end, - mnesia:dirty_write(HashedRecord) + mnesia:write(HashedRecord) end. lookup(Username) -> case mnesia:dirty_read(ts_user, Username) of [] -> no_record; [User] -> User - end. + end. list(Start, Length) -> ts_common:list(ts_user, Start, Length).