Not all sensitivities are true allergies that will lead to anaphylaxis and death. For example, most pollen "allergies" are really sensitivities that do not cause more than itchy eyes and runny noses and do not lead by repeated exposures to more serious reaction. When you work walnut do you have breathing problems? Do you develop hives? Do you get nauseated? Feel dizzy or lightheaded?
Certainly if you are concerned about it you can get tested by an allergist, but more than likely your mild symptoms can be pre-treated with a non-sedating antihistamine such as Claritin or Zyrtec.