View source for Ripping data from refractiveindex.info for nk function

= README — Building and Maintaining the '''nk-data''' Database =

This page explains how to (1) recreate the '''nk-data''' folder (CSV files + catalog) from the refractiveindex.info (RII) YAML database, and (2) add custom sources manually so the shared '''nk''' function can use them immediately. 

(Let me know of any glaring errors in the README as I haven't scrutinised it nearly as much as I did the code it's talking about.)

''TL;DR:'' Run <code>rii2catalog.py</code> once to build <code>nk-data/</code>. When you get a new dataset, drop a CSV into <code>nk-data/data/</code> and add a small JSON block to <code>nk-data/catalog/catalog.json</code>.

----

== 0) Expected layout ==

<pre>
project/
├─ nk.m                    % (or nk_sam.m) the MATLAB function
└─ nk-data/
   ├─ data/                % CSV files: wavelength_um,n,k
   └─ catalog/
      └─ catalog.json      % all sources + metadata
</pre>

The function looks for <code>nk-data/</code> next to itself (or via <code>opts.data_root</code> or the <code>NK_DATA_ROOT</code> environment variable).

----

== 1) Rebuild '''nk-data''' from refractiveindex.info ==

=== Prerequisites ===
* Python 3.9+  
* PyYAML (for parsing RII YAML)  
* The RII database (release zip or cloned repo)

PowerShell:
<pre>
python --version
pip install pyyaml
</pre>

=== Get the RII database ===
Download/unzip the official RII database so you have a path like:
<pre>
C:\...\rii-db\database\
  data\main\...
  data\organic\...
  data\glass\...
  data\other\...
  data\sopra\...
  data\3d\...
</pre>
(Those subfolders are “shelves” of materials.)

=== Run the converter ===
Use the included <code>rii2catalog.py</code> to convert YAML → CSV + catalog JSON.

PowerShell (from your project folder that has <code>nk.m</code> and <code>rii2catalog.py</code>):
<pre>
$DBROOT = "C:\Users\&lt;you&gt;\...\rii-db\database"
python .\rii2catalog.py --db-root "$DBROOT" --out-root .\nk-data --log-sampling `
  --shelves data/main data/organic data/glass data/other data/sopra data/3d
</pre>

What it produces:
* CSV files into <code>nk-data/data/</code> with the exact header:
  <pre>wavelength_um,n,k</pre>
  Wavelengths are in '''micrometers''' (µm). <code>n</code> or <code>k</code> may be blank if absent.
* <code>catalog.json</code> into <code>nk-data/catalog/</code> with, per source:
  * <code>source_id</code> (stable identifier)
  * <code>kind</code> (<code>thinfilm</code> or <code>bulk</code>) and <code>valid_thickness_nm</code> if detected
  * <code>path</code> (relative to <code>nk-data</code>)
  * <code>valid_wavelength_um</code> = <code>[wmin, wmax]</code>
  * <code>has_k</code> (boolean)
  * <code>year</code> (parsed from the reference text; newer wins ties)
  * <code>ref_string</code>, <code>notes</code>, provenance fields
  * SHA-256 checksums for each CSV (optional but included)

Supported formula sampling:
* Formula '''1''' (Sellmeier-like) and '''4''' (Cauchy) → sampled to n(λ)
* Other formula types are skipped (console shows <code>[SKIP]</code>)

'''Notes'''
* All wavelengths are stored in µm (RII’s internal unit). The MATLAB caller can pass meters/nm/µm; <code>nk</code> converts internally.
* Pages with multiple DATA blocks (e.g., multiple thicknesses) produce multiple CSVs. Per-entry thickness is inferred where possible.
* Some families (e.g., ITO) live under umbrella groups in RII. The MATLAB function augments these at runtime (e.g., synthetic <code>materials.ito</code> built from <code>mixed_crystals</code>).

----

== 2) Manually adding a new dataset ==

You can add lab data or literature without re-running the scraper.

=== 2.1 Prepare the CSV ===
Create a file in <code>nk-data/data/</code> with this exact header:
<pre>wavelength_um,n,k</pre>
* Wavelengths: in '''µm''', strictly increasing.
* If you only have '''n''', leave the <code>k</code> column blank. The MATLAB function can treat missing k as zero with <code>opts.allow_k_zero = true</code>.
* Suggested filenames: <code>&lt;material&gt;.&lt;page&gt;.&lt;tag&gt;.csv</code>, e.g.:
  <pre>au.smith-2025.d1.csv
sio2.custom-lab-2025.csv</pre>

=== 2.2 Add a source entry to <code>catalog.json</code> ===
Open <code>nk-data/catalog/catalog.json</code>. Find the target material under <code>materials</code>. If it doesn’t exist, add a new object for it. Append a source record:

<pre>{
  "source_id": "au.smith-2025.d1",
  "kind": "thinfilm",                      // or "bulk"
  "path": "data/au.smith-2025.d1.csv",     // relative to nk-data/
  "valid_wavelength_um": [0.4, 1.1],       // µm
  "valid_thickness_nm": [100, 100],        // [tmin,tmax] in nm, or null for bulk
  "priority": 1,                           // not used by nk; keep 1
  "year": 2025,                            // tie-breaker (newer wins)
  "ref_string": "Smith et al., Opt. Lett. (2025) ...",
  "notes": "Ellipsometry, sputtered Au on glass, room temp.",
  "shelf": "lab",
  "book": "Au",
  "page": "smith-2025-d1",
  "has_k": true
}</pre>

Material block (simplified):
<pre>"au": {
  "aliases": ["Au","gold"],
  "default_source_id": null,
  "sources": [
    { ... },                               // existing sources
    {
      "source_id": "au.smith-2025.d1",
      "kind": "thinfilm",
      "path": "data/au.smith-2025.d1.csv",
      "valid_wavelength_um": [0.4, 1.1],
      "valid_thickness_nm": [100, 100],
      "priority": 1,
      "year": 2025,
      "ref_string": "Smith et al., Opt. Lett. (2025) ...",
      "notes": "Ellipsometry, sputtered Au on glass, room temp.",
      "shelf": "lab",
      "book": "Au",
      "page": "smith-2025-d1",
      "has_k": true
    }
  ]
}</pre>

Checksums (optional, for <code>info.data_checksum</code>):
<pre>
$h = (Get-FileHash .\nk-data\data\au.smith-2025.d1.csv -Algorithm SHA256).Hash.ToLower()
</pre>
Add to the <code>"checksums"</code> object:
<pre>
"data/au.smith-2025.d1.csv": "sha256-<paste the hash here>"
</pre>

=== 2.3 Field rules (quick) ===
* <code>source_id</code>: unique; recommend <code>&lt;material&gt;.&lt;page&gt;[.dN]</code>
* <code>kind</code>: <code>"thinfilm"</code> if thickness known, else <code>"bulk"</code>
* <code>valid_wavelength_um</code>: matches your CSV’s min/max
* <code>valid_thickness_nm</code>: <code>[t,t]</code> for single thickness; <code>null</code> for bulk/unknown
* <code>year</code>: 4-digit year (tie-breaker)
* <code>has_k</code>: <code>true</code> if the k column exists (even if some rows are blank)

----

== 3) Sanity checks in MATLAB ==

Point <code>nk</code> at your database if it isn’t next to <code>nk.m</code>:
<pre>
setenv('NK_DATA_ROOT', 'C:\path\to\project\nk-data')
</pre>

Examples:
<pre>
% Thin-film preference (e.g., Ag at 532 nm, 100 nm film)
[n,k,info] = nk(532e-9, 'ag', struct('units','m','thickness_nm',100));
disp(info.source_id); disp(info.selection_reason);

% Bulk fallback (thickness > 500 nm)
[n2,k2,info2] = nk(780e-9, 'au', struct('units','m','thickness_nm',600));

% Legacy names and inline paths
[n3,k3,info3] = nk(532e-9, 'fused silica', struct('units','m'));  % → SiO2
[n4,k4,info4] = nk(532e-9, 'BK7', struct('units','m'));          % inline Sellmeier
[n5,k5,info5] = nk(532e-9, 'ITO', struct('units','m'));          % ITO aggregator
</pre>

Expected behavior:
* For thickness ≤ 500 nm, thin-film datasets are preferred (closest thickness).  
* Tie-breaks: '''has k''' → '''newer year''' → '''wider λ-span''' → <code>source_id</code>.  
* Legacy names (e.g., <code>water</code>, <code>fused silica</code>, <code>diamond</code>, <code>air</code>, <code>ITO</code>) resolve.  
* <code>BK7</code> uses the inline Sellmeier to preserve legacy results.

----

== 4) Troubleshooting ==

* '''“No source covers …”''' – The wavelength must be inside a source’s <code>[λmin, λmax]</code>. The error lists the top partial overlaps; choose another wavelength or add a source.
* '''“CSV missing column ‘…’”''' – The CSV header must be exactly <code>wavelength_um,n,k</code>. Units are µm.
* '''“Chosen source lacks k-values”''' – Either add k to the CSV or call with <code>opts.allow_k_zero = true</code>.
* '''Legacy name not found''' – Add it to the alias table in <code>nk.m</code> (function <code>local_alias_map()</code>). For umbrella cases (e.g., ITO), the function augments the catalog at runtime.
* '''Default units''' – If old code passed meters, set the default in <code>local_parse_opts</code>:
  <pre>'units','m'   % instead of 'um'</pre>

----

== 5) Conventions & credits ==

* '''Wavelength unit:''' all CSVs store wavelengths in µm. Callers can use <code>opts.units</code> (<code>'m'</code>/<code>'nm'</code>/<code>'um'</code>).  
* '''Thickness unit:''' nm in the catalog.  
* '''Credit:''' Data derived from refractiveindex.info; cite the original publications listed in <code>ref_string</code>.

----

== 6) One-line rebuild (for future you) ==
<pre>
$DBROOT = "C:\...\rii-db\database"
python .\rii2catalog.py --db-root "$DBROOT" --out-root .\nk-data --log-sampling `
  --shelves data/main data/organic data/glass data/other data/sopra data/3d
</pre>