Over the last years, the trend associated with the incorporation of materials from renewable resources into polymer technology is getting significantly more vital. Researchers are trying to transfer the properties of natural raw materials into the polymer world. Therefore, different natural materials are more often investigated as potential additives for polymers. Such an effect is noted for the coffee industry by-products, such as coffee silverskin. Because of the relatively high contents of compounds showing antioxidant activity, such as caffeine, polyphenols, tannins, or melanoidins, this by-product could be considered not only as a filler, but also as a potential modifier for polymer materials. Its antioxidant activity is comparable to commercially available antioxidants applied in polymer technology. Therefore, in the presented paper, we examined the influence of the coffee silverskin (from 1 to 20 wt%) on the thermal and mechanical performance of polyethylene-based composites. Moreover, materials were subjected to accelerated aging tests in the UV chamber, which revealed that coffee silverskin could inhibit the photodegradation of the polymer matrix. Therefore, this by-product should be considered as an exciting alternative for the conventional lignocellulosic fillers, which could provide additional features to polymer composites.