dm: clk: ignore default settings when node not valid

When the device not binded with a node, we need ignore
the parents and rate settings.

Cc: Simon Glass <sjg@chromium.org>
Cc: Jagan Teki <jagan@amarulasolutions.com>
Cc: Philipp Tomsich <philipp.tomsich@theobroma-systems.com>
Cc: Neil Armstrong <narmstrong@baylibre.com>
Cc: Andreas Dannenberg <dannenberg@ti.com>
Signed-off-by: Peng Fan <peng.fan@nxp.com>
diff --git a/drivers/clk/clk-uclass.c b/drivers/clk/clk-uclass.c
index 85dfe71..cee4d91 100644
--- a/drivers/clk/clk-uclass.c
+++ b/drivers/clk/clk-uclass.c
@@ -285,6 +285,9 @@
 {
 	int ret;
 
+	if (!dev_of_valid(dev))
+		return 0;
+
 	/* If this not in SPL and pre-reloc state, don't take any action. */
 	if (!(IS_ENABLED(CONFIG_SPL_BUILD) || (gd->flags & GD_FLG_RELOC)))
 		return 0;