Translate

Showing posts with label SQL. Show all posts
Showing posts with label SQL. Show all posts

Sunday, April 26, 2015

Monitoring health data - populating Food tables

This article continue the discussion about the building of the system to collect my health numbers.

There are already many data stored in LibreOffice Calc tables and other LibreOffice documents.
Because of that when/where possible macro will be used to generate the mySQL commands to populate the tables.

Populate T_FOOD_TYPE

Populating this table it requires more work because it is necessary to find the information about each specific food used. In order to simplify the input mechanism, a new spreadsheet that collect the information to be put in the database, is created.
Unfortunately there is no easy way to populate the spreadsheet, i.e. it must be done manually, copying the data from the existing LibreOffice word documents as described in the Monitoring health data - my way  #2 article.

Let's examine the spreadsheet used to format the insert statement.

The spreadsheet has two tabs:
  • Food type
  • Food
The Food type tab, is structured as the database, raw.
A hash key is prepared as index, using the Food name, the brand and the number of carbs.
This should allow to generate a unique key for the each record.

Here the fields of the tab:

Index (food type) Name 1 Serving Serving unit Serv X cont Carbs Calories Protein Fat Sodium Potassium Brand Notes


The index in the cell A2 use the Hash macro (it is implied the third Party Hash macro is installed):

=TEXTHASH(CONCATENATE(B2,L2,F2),"MD5")

The goal is to have a unique index and to do so the hash is calculated using the name of the food (B2), the value in carbs (F2) and the brand (L2).
The field in the table are mostly following the SQL table structure (see Monitoring health data - an improvement).
For extra info there are these fields:
  • Serv X cont
    Serving per container - the number of serving you can expect from the container or how many servings can be obtained from a recipe
  • Notes
    Just extra notes about the food
After the table, there is the formula for the formatting of the SQL insert statement.
First of all, a cell (O1) contains the starting structure of the statement :

Insert into t_food_type values (

Then below, this formula is applied :
=IF(B2="","",(CONCATENATE($O$1,"""",A2,"""", ",", """", B2, """", ",", C2, ",", """", D2, """", ",", F2, ",", G2, ",", H2, ",", I2, ",", J2, ",", K2, ",",  """", L2, """", ",", """", M2, """", ");")))
The meaning is :
  • if the field "Name" exists (not blank) then 
    • create a string using the field O1, plus the other fields, starting from the field A2 (hash key) and ending with the field M2 (Notes)
  • otherwise create a blank entry
The formula is then copied in all subsequent cells.

Here an example of the insert statement generation starting with the data on the table :

Insert into t_food_type values ("ce7993e549b4d010c825766dfd034a99","Pork Patties",2,"patty",2,270,11,24,500,0,"Jimmy Dean","");
Insert into t_food_type values ("981a21f5a752cad048a834b9fd6cd258","Cooked Ham",4,"slices",1,60,8,2,580,0,"Land O'Frost","");
Insert into t_food_type values ("1a52451ea65def964ddb3c2be50811ce","Big Cup Noodles Homestyle Beef Flavor Ramen Noodle",1,"1/2 container",24,190,3,8,760,0,"Nissin","");
Insert into t_food_type values ("74ccbb6a8f0e507375ca5f727074a4e1","Texas Style Angus Beef Chili with Black Beans Soup",1,"cup",21,210,15,5,790,0,"Campbell","");
Insert into t_food_type values ("76d5f224ea4a78e3d737caa4eba0d1e","iesta Chicken Lime Tortilla Soup with White Meat Chicken",1,"cup",13,110,9,1.5,790,0,"Campbell","");
.............................................................
.............................................................

After the table is manually populated, we end up with a list of insert capable to fill up the T_FOOD_TYPE table.

Populate T_FOOD

The same mechanism used  to populate the T_FOOD_TYPE table, is used to populate the T_FOOD table too.
Like before, a new spreadsheet that collect the information to be put in the database is created (still refer to the Monitoring health data - my way  #2 article)

Let's examine the spreadsheet used to format the insert statement.

The spreadsheet has two tabs:
  • Food type
  • Food
The Food tab, is structured as the database, raw.
Here the table structure:

Index Date Meal Type Meal FoodIndex Food name Servings Calculated carbs Calculated salt

The Calculated carbs and Calculated Salt are populated using data from the previous table.

Calculated carbs is :

=VLOOKUP(D2,'Food type'.$A$2:$M$156, 6, 0)*F2

The meaning is to look in the all previous tab (Food type) of the spreadsheet for the FoodIndex key, retrieve the number of carbs per serving and multiply for the number of serving (field Servings on this tab).

Similarly for the Calculated salt :

=VLOOKUP(D2,'Food type'.$A$2:$M$156, 10, 0)*F2

The meaning is to look in the all previous tab (Food type) of the spreadsheet for the FoodIndex key, retrieve the number of mg of salt per serving and multiply for the number of serving (field Servings on this tab).
Just to have a quick idea about the amount of salt of the selected food.

The Index is an hash key using the Date Meal, the FoodIndex and the Food name fields. 
This should allow to generate a unique key for the each record.
The cell A2 contains this formula :

=TEXTHASH(CONCATENATE(B2,C2,D2,E2),"MD5")

Again the purpose is to obtain a unique key for each entry, to do so it used the date of the meal (B2), the type of the meal (C2, e.g. Breakfast or Lunch or Dinner or Snack), the food index (D2, foreign key) and the food name (E2).
Note that the food name field is populated from the other Calc table (see above).

After the table, there is the formula for the formatting of the SQL insert statement.
First of all, a cell (I1) contains the starting structure of the statement :

Insert into t_food values (

Then below, this formula is applied :

=IF(B2="","",(CONCATENATE($H$1, """", A2, """", ", str_to_date('",(TEXT(B2,"mm/dd/yy")), "', ", """", "%m/%d/%Y", """", "), ",        """",C2,"""",", ","""",D2,"""", ", " ,F2, ");")))


The meaning is :
  • if the field "Date Meal" exists (not blank) then 
    • create a string using the field H1, plus the other fields, starting from the field A2 (hash key) and ending with the field F2 (Servings)
  • otherwise create a blank entry
The formula is then copied in all subsequent cells.

Here an example of the insert statement generation starting with the data on the table :

Insert into t_food values (84280b208c8c61a7aab5c4be5fb25385, str_to_date('12/29/14', "%m/%d/%Y"), "Breakfast", "ac118ebecb401ae2e9fe42b8f7b1da8c", 2);
Insert into t_food values (16df74f3237c526a71186884b0cd7aa1, str_to_date('12/29/14', "%m/%d/%Y"), "Breakfast", "254a86d2153bfbc149f51d63c7113952", 1);
Insert into t_food values (532061254f4761e69bcf627c4916f3c2, str_to_date('12/29/14', "%m/%d/%Y"), "Lunch", "1b4f22d0d92b9185406cf6d815fd2c4c", 1);
Insert into t_food values (34c6041c3126488e599aa467e8a0d4de, str_to_date('12/29/14', "%m/%d/%Y"), "Lunch", "af3a803a27460603422bf47322ba6d4", 2);
Insert into t_food values (532061254f4761e69bcf627c4916f3c2, str_to_date('12/29/14', "%m/%d/%Y"), "Dinner", "1b4f22d0d92b9185406cf6d815fd2c4c", 2);
....................................................................
....................................................................


Basically after the table is manually populated, we end up with a list of insert capable to fill up the T_FOOD table.


Saturday, April 25, 2015

Monitoring health data - populating measurement tables

This article continue the discussion about the building of the system to collect my health numbers.

There are already many data stored in LibreOffice Calc tables and other LibreOffice documents.
Because of that when/where possible macro will be used to generate the mySQL commands to populate the tables.
The T_MES_TYPE table will be populated manually, but T_MES, T_FOOD_TYPE and T_FOOD can use some kind of automation mechanism to do so.

Populate T_MES_TYPE

Here a list of mySQL commands to populate the T_MES_TYPE table.
It is easy to populate the table, just open a mySQL session in a terminal, connect to the database and then cut & paste the lines below.

connect name_database;
insert into t_mes_type values (1, "Glicemy", "ml/dg");
insert into t_mes_type values (2, "Weight", "lb");
insert into t_mes_type values (3, "Carbs", "count");
insert into t_mes_type values (4, "Blood Systolic", "mmHg");

insert into t_mes_type values (5, "Blood Diastolic", "mmHg"); 
insert into t_mes_type values (6, "Heart beat", "bpm"); 

insert into t_mes_type values (7, "Ketons", "count"); 
insert into t_mes_type values (8, "Exercise elliptical", "min"); 
insert into t_mes_type values (9, "Exercise walking", "miles"); 
insert into t_mes_type values (10, "Exercise hiking", "miles"); 

Populate T_MES table from Spreadsheet

Since already exists many data stored in the LibreOffice spreadsheet, here there is  a procedure to populate the data stored in the tables. 

Populate T_MES - glucose level


  • Open the spreadsheet
  • Create a separate tab and rename it to give a meaning (i.e. Export SQL glucose)
  • In the cell A1 copy this fixed string : Insert into t_mes values ( 
  • In the cell J2 use the Hash macro (it is implied the third Party Hash macro is installed):

    =TEXTHASH(CONCATENATE('All times'.A2,'All times'.F2,'All times'.G2),"MD5")
  • Copy in the other Jx fields the formula above
  • In the cell A2 copy this formula :

    =IF (OR('All times'.G2 = 0, 'All times'.G2 = ""), "",(CONCATENATE($A$1, """", J2,"""", ", str_to_date('", (TEXT('All times'.A2,"mm/dd/yy")), "',", """", "%m/%d/%Y", """", ")", ", str_to_date('", (TEXT('All times'.F2,"hh:mm:ss")), "',", """", "%H:%i:%s", """", ")", ", 1, ", 'All times'.G2, ");")))
  • Copy all the result strings  in the mysql
Here an example of a correct entry generated :

Insert into t_mes values ("31ba5af2e19e9c8e9bc35bbd1f588fb1", str_to_date('10/28/14',"%m/%d/%Y"), str_to_date('19:45:00',"%H:%i:%s"), 1, 150);

Invalid entries will be blanket

Populate T_MES - weight


  • Open the spreadsheet
  • Create a separate tab and rename it to give a meaning (i.e. Export SQL weight)
  • In the cell A1 copy this fixed string : Insert into t_mes values ( 
  • In the cell J2 use the Hash macro (it is implied the third Party Hash macro is installed):

    =TEXTHASH(CONCATENATE('All times'.A2,'All times'.F2,'All times'.H2),"MD5")
  • Copy in the other Jx fields the formula above
  • In the cell A2 copy this formula :

    =IF ( OR ('All times'.H2 = 0, 'All times'.H2 = ""), "", (CONCATENATE($A$1, ,"""",J2,"""", ", str_to_date('", (TEXT('All times'.A2,"mm/dd/yy")), "',", """", "%m/%d/%Y", """", ")", ", str_to_date('", (TEXT("08:00:00","hh:mm:ss")), "',", """", "%H:%i:%s", """", ")", ", 2, ", 'All times'.H2, ");")))
  • Copy the content of the cell A2 for the remains rows up tp the number of rows present in the tab All times (see the limit in the tab All times)
  • Copy all the result strings  in the mysql
Here an example of  a correct entry generated :

Insert into t_mes values ("4af7c8660faee64b3e126c3c1717b510", str_to_date('10/28/14',"%m/%d/%Y"), str_to_date('08:00:00',"%H:%i:%s"), 2, 177.6);


Invalid entries will be blanketed

Populate T_MES - Ketons


  • Open the spreadsheet
  • Go in the tab of the month (to be repeated for each month)
  • In the cell U1 copy this fixed string : Insert into t_mes values ( 
  • In the cell T2 column create the index using the hash formula:

    =TEXTHASH(CONCATENATE(A2,F2,K2),"MD5")
  • Copy in the other Tx fields the formula above
  • In the cell U2 copy this formula :

    =IF (   K2 = "",    "",       (CONCATENATE($U$1,                 """",T2,"""",         ", str_to_date('", (TEXT(A2,"mm/dd/yy")), "',", """", "%m/%d/%Y", """", ")",         ", str_to_date('", (TEXT("09:00:00","hh:mm:ss")), "',", """", "%H:%i:%s", """", ")",         ", 7, ", K2, ");")))
  • Copy the content of the cell U2 for the remains rows up tp the number of rows present in the monthly tab 
  • Copy all the result strings  in the mysql
Here an example of a generated entry (invalid entries are blanketed).

Insert into t_mes values ("fe8565c8659775e73957e9c0485101f4", str_to_date('03/01/15',"%m/%d/%Y"), str_to_date('09:00:00',"%H:%i:%s"), 7, 0);

Alternatively (I prefer this way) is possible to prepare ASCII files with the generated commands and feed mySQL with these files.
In this way, the files allows to recreate the database from the scratch if necessary, a form of backup, and also is easy to duplicate the database on different machines.
The files can be edited to add comments and explain the purpose of specific commands/fields.

Sunday, April 5, 2015

Monitoring health data - an improvement

The purpose of the article is to start a discussion  about a "system" capable to store and retrieve/analyze health data.
More articles will follow on specific issues.  This article is the continuation of the previous articles "Monitoring health data - my way #1 and #2".

The use of the spreadsheet as main tool to collect data for now is preserved, because it is  easy to store the information in that way, but retrieve them in a meaningful way can become tricky.
This why I started to think a better way to do this "daily chore", in a way that will easily allow me to create better reports and analyze the data more deeply, creating also reports "on the fly".

The system


The system is based on a database that collect all the information and a set of macro/applications/programs to facilitate the data entry and the retrieval.
The database is based on mySQL.
Macro in the spreadhseet (Libreoffice Calc) allows to export the data stored in the SQL tables.

The database


The database is based on some tables that collects the measurements and other related values.
At the moment I created 4 tables:

  • Measurement table
  • Measurement type table
  • Food table
  • Food type table

Measurement tables

T_MES


These are the main tables used to store the health measurements.
The main table (T_MES) contains the actual readings, and the T_MES_TYPE contains specific information about the type of measurement.
Each measure is stamped with date and time, however some measurements will have a fake time.
For example the weight. Usually there is an entry per day, so the time is forced at 8:00 am regardless when actually was taken, because not always I can measure at the same time.

Let see in details the fields of T_MES:

  • mesIndex
    Unique ID for the record, it is a MD5 Hash of the date, time and value measured
  • dateMeasure
    It is the date of the measurement
  • timeMeasure
    It is the time of the measurement. Weight and Ketons have a fake time entry. 8:00 am for Weight, 9:00 am for Ketons
  • typeMeasure
    Foreign key for the table T_MES_TYPE
  • noteMeasure
    Contains specific measurement note. The nature of the note depends about the measurement.
  • valueMeasure
    Value of the measurement
The T_MES_TYPE table contains information about the measure.
  • typeMeasure
    Unique ID for the record. Incremental number
  • description
    A brief description of the measurement. Explains what is measured
  • units
    Measurement unit of the measurement

Here the data to load in the T_MES_TYPE table :

typeMeasure code unitsdescription 
1ml/dg Glycemy
2lb  Weight 
3 countCarbs 
4mmHgBlood pressure systolic 
5 mmHg Blood pressure diastolic
6 bpmHeart beat (beat per minute)
 7count
Ketons 
 8minutes
Exercise elliptical 
 9miles 
Exercise walking
 10miles 
Exercise hiking

Here the mySQL commands to create the two tables :

create table if not exists t_mes (mesIndex varchar(40) not null primary key, dateMeasure date, timeMeasure time, typeMeasure numeric, valueMeasure numeric);

create table if not exists t_mes_type (typeMeasure numeric not null primary key, description varchar(20), unit varchar(10));

T_FOOD


The T_FOOD tables (T_FOOD and T_FOOD_TYPE) stores information about the food eaten.

Let see in details the fields of T_FOOD:

  • foodIndex
    Unique ID for the record, it is a MD5 Hash of the date, time and value measured
  • dateMeal
    It is the date of the meal
  • typeMeal
    It is the time of the meal according to a fixed schedule.
    It is a fixed value like this :
    • Breakfast
    • Lunch
    • Dinner
    • Snack
  • foodType
    Foreign key for the table T_FOOD_TYPE
  • serving
    Number of serving consumed
The T_FOOD_TYPE table contains information about the food.
  • foodType
    Unique ID for the record. Hash of the food name, brand and carbs
  • name
    Name of the food
  • servings
    Number of servings in a product (if applicable)
  • servingUnits
    Measurement unit used for the servings
  • carbs 
    Net Carbs in grams contained in 1 serving
  • calories
    Calories contained in 1 serving
  • protein
    Proteins in grams contained in 1 serving
  • totalFat
    Total fat in grams contained in 1 serving
  • sodium
    Salt in milligrams contained in 1 serving
  • potassium
    Potassium in milligrams contained in 1 serving
  • brand
    Name/description brand of the food. For vegetables/fruit use the word "raw".
  • notes
    Notes about the food
Here the mySQL commands to create the two tables :

create table if not exists t_food (foodIndex varchar(40) not null primary key, dateMeal Date, typeMeal varchar(30), foodType varchar(40), serving Numeric);


create table if not exists t_food_type( foodType varchar(40) primary key not null, name varchar(40), servings Float, servingUnits varchar(15), carbs Float, calories Float, protein Float, totalFat Float, sodium Float, potassium Float, brand varchar(40), notes varchar(40));